Orange pi zero kernel oops with zram enabled

[atomic77]

Hi all,

 

I've been doing some stress tests to trigger the watchdog on an Orange Pi Zero and I discovered that i can reliably reproduce a kernel oops with a simple fork bomb on a fresh install of the latest Armbian Buster image (Armbian_20.11_Orangepizero_buster_current_5.8.16.img.xz)

Spoiler

[  118.737219] Internal error: Oops: 5 [#1] SMP THUMB2
[  118.742106] Modules linked in: xradio_wlan mac80211 btusb btintel btrtl sun4i_gpadc_iio btbcm industrialio bluetooth sun8i_thermal cfg80211 ecdh_generic rfkill ecc libarc4 sun8i_ce crypto_engine sunxi_cedrus(C) sun8i_di zram uio_pdrv_genirq uio cpufreq_dt usb_f_acm u_serial g_serial libcomposite ip_tables x_tables autofs4 pwrseq_simple sunxi phy_generic
[  118.773408] CPU: 3 PID: 3792 Comm: bash Tainted: G         C        5.8.16-sunxi #20.08.14
[  118.781661] Hardware name: Allwinner sun8i Family
[  118.786372] PC is at obj_malloc+0x28/0xf0
[  118.790378] LR is at zs_malloc+0x149/0x364
[  118.794469] pc : [<c0262500>]    lr : [<c0262c4d>]    psr: 20010133
[  118.800725] sp : c890b880  ip : cdf4a208  fp : cdf4a201
[  118.805942] r10: 1bc20381  r9 : 4a5ff920  r8 : d4bac480
[  118.811159] r7 : cc318fc0  r6 : 0004a5ff  r5 : 00000000  r4 : 00000000
[  118.817677] r3 : 00000004  r2 : 00000000  r1 : cc318fc0  r0 : d4bac480
[  118.824197] Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA Thumb  Segment none
[  118.831495] Control: 50c5387d  Table: 4b1e406a  DAC: 00000051
[  118.837234] Process bash (pid: 3792, stack limit = 0x064576e6)
[  118.843058] Stack: (0xc890b880 to 0xc890c000)
[  118.847414] b880: 00000001 d4bac480 cdf4a200 c9d1b800 0000280a 00002800 cdf4a200 cc318fc0
[  118.855584] b8a0: bf85f440 c0262c4d d6adc000 00000001 d7bf6d30 c890b8f8 00002000 c04efee1
[  118.863752] b8c0: d6adc000 c0f04fc8 000008d9 d48f0c00 00000000 c890a000 d7928dd0 00000000
[  118.871921] b8e0: ff7f6760 c100247c bf85f440 bf85ccd3 00000ed7 00000001 000008d9 c0f04fc8
[  118.880089] b900: ffff4ef6 000076b8 00000000 d48f0c00 d65edf80 00000001 d7928dd0 00000000
[  118.888257] b920: ffff4ef6 bf85d22d 00000000 c029f57b d7928dd0 00001000 00000000 c0f04fc8
[  118.896425] b940: ffffffff 00000000 d65edf80 000076b8 00000000 d7928dd0 bf85e064 00000001
[  118.904593] b960: c890bad8 c029f5a9 d7928dd0 00000001 38e38e39 d7928dd0 d65edf80 c890ba48
[  118.912762] b980: c024745d d5689e00 c890bae0 c02477c7 d7928dd0 c890ba48 00000000 c024d489
[  118.920930] b9a0: c10323a8 c27be0c0 00001000 d7928dd0 00000ed7 c0f04fc8 cc5c89d8 d7928dd0
[  118.929099] b9c0: c0f28fe4 d7928dd0 00000000 c0f055f4 d52503c0 00000ed7 c890bad8 c024d489
[  118.937267] b9e0: d7928dd0 c890ba48 00000000 c0f04fc8 d5689e00 d7928dd4 d7928dd0 00000000
[  118.945435] ba00: c890bc64 d5689e00 c890bae0 c02172b7 00000001 00000000 ffffe000 00000000
[  118.953603] ba20: 00000003 00000000 c0fd71c0 c0e961cc 00000380 00000001 c890ba38 c890ba38
[  118.961771] ba40: d792cc20 d791ca80 00000020 00000000 00000000 00000000 ffffffff 7fffffff
[  118.969938] ba60: 00000000 00000008 00000000 00000000 00000000 00000000 00000000 00000000
[  118.978106] ba80: 00000000 00000000 00000000 c0f04fc8 c0f04fc8 c890bc64 c0e96180 d6267000
[  118.986275] baa0: c0fd71c0 00000020 00000000 00000007 c0fd7cc0 c0218217 c890bae0 00000000
[  118.994443] bac0: c890bc64 00000003 fffffffe 00000068 00000000 00000020 d7bdc808 d792a25c
[  119.002610] bae0: 00000000 00000000 00000000 00000000 00000000 00000003 00000000 00000000
[  119.010778] bb00: 00000000 00000000 00000000 c0f04fc8 c021878f c890bc64 d6267000 00000020
[  119.018946] bb20: 00000000 00000000 51eb851f 00000000 c890bb70 c0218983 0001b06e ffffffff
[  119.027115] bb40: 000001ee 00000000 e0802000 00000020 000001ed 00000000 c890bb58 c890bb58
[  119.035283] bb60: c890bb60 c890bb60 80000000 00000031 00000045 00000001 00000001 c0193615
[  119.043451] bb80: 00000051 00000045 00000001 00000001 c0193615 c0f04fc8 00000000 c890bc64
[  119.051619] bba0: cd6ef800 00000000 c0f05604 00000000 0000000e 00000028 c0fd71c0 c0218c7d
[  119.059787] bbc0: 00000000 c0fd71c0 d7006e00 00000000 0000001f 00000000 c0f04ac0 c0f15808
[  119.067956] bbe0: c0fd7cc0 c890bc8c c890bca8 c0f04a80 00400dc2 c890bc64 c0fd7c40 c0fd71c0
[  119.076124] bc00: 00000000 00400dc2 c0fd7c40 00000000 c1035d74 c02190ff 00100cc0 00200010
[  119.084292] bc20: c890a000 0000000c c890bc18 c0e96180 00000000 c0f04fc8 00000840 00000001
[  119.092460] bc40: 00400dc2 00000000 00000000 c0fd7c40 c890a000 00400dc2 00000000 c0219d11
[  119.100628] bc60: 00000001 00000020 00000000 00000000 000021bc 00000170 0600007a 00000001
[  119.108796] bc80: 00400dc2 00000012 00000028 00000001 00000001 00000000 00000000 00000000
[  119.116964] bca0: 0000000b 0000000e 00000000 c0f04fc8 00400140 cd67af40 00400940 00000000
[  119.125133] bcc0: c0f054d0 c0243729 dff969c0 dff95ae4 dff96f78 00000001 c0792aa1 00000840
[  119.133301] bce0: 00000000 00000000 00400dc2 00000000 dff969c0 cc55e5c0 16953a8c 00000840
[  119.141469] bd00: 16421569 00000001 00000000 00000400 00000000 00000000 005009c2 00000000
[  119.149637] bd20: c0f054d0 00000000 00000000 00000840 00000810 00400dc2 00000010 00000850
[  119.157805] bd40: 00000049 c0146567 c0fd7c40 00000000 c0fd7c40 00000000 00000001 00000000
[  119.165973] bd60: cc55e5c0 c013d8ef c0a025c0 c0a02624 dff96980 c890bd80 20000113 c013d8ef
[  119.174142] bd80: c890bdd4 c0f04fc8 cc55e540 c8d9e200 c8d9e200 cc728010 004f7000 d5accaa8
[  119.182310] bda0: 004f7000 c890a000 c8a3bde0 c0230273 cc728010 c8d9e200 cc728010 00000000
[  119.190479] bdc0: 004f7000 c0230d7d c890bde0 cb1e4018 cc728010 004f9000 00000000 c0f04fc8
[  119.198647] bde0: c8b3be04 cc55e9e0 004f9000 c81ae360 c890be08 00000000 c8a3bdf0 d5accac8
[  119.206815] be00: d6330200 004f8fff d5accac8 00000000 00000000 00000000 00000000 c0f04fc8
[  119.214983] be20: c8a3ba20 c8a3ba20 c8d9e200 d6330200 c81ae360 d5accaa8 c8a3ba30 d5accac8
[  119.223152] be40: c8a3bde0 c011949b c15d4340 c0254a7f c8a3ba28 c8a3ba34 c890a000 d6330240
[  119.231320] be60: c8d9e240 c100257c c1706500 c890be6c c890be6c c0f04fc8 c1f1fa5c 01200000
[  119.239488] be80: cc55ca40 00000000 ffffe000 c890a000 c6514c00 00000000 00000000 c011a759
[  119.247656] bea0: d6330240 00000800 00000255 c890bf58 00000000 00000000 00000000 00000000
[  119.255825] bec0: 00000000 00000000 c100257c cc55cd54 00000000 ffffe000 fffffff4 00000000
[  119.263993] bee0: 00000000 00000000 c175bc34 c0f04fc8 00000cc0 b6ff5238 00000000 c890bf58
[  119.272161] bf00: 00000078 c0100284 c890a000 00000078 01200000 c011ab5f cd67b37c c0126d4b
[  119.280329] bf20: c890bfb0 01e176e4 0000080f c0f04fc8 d6330200 b6ff5238 00101000 00000000
[  119.288498] bf40: 00000078 c0100284 c890a000 00000078 01e2b7c8 c011b07f 01200000 00000000
[  119.296665] bf60: 00000000 b6ff5238 00000000 00000011 00000000 00000000 00000000 00000000
[  119.304833] bf80: 00000000 00000000 00000000 00000000 b6ff6968 c0f04fc8 00000000 b6ff5238
[  119.313001] bfa0: b6ff5690 c0100061 b6ff5238 b6ff5690 01200011 00000000 00000000 00000000
[  119.321169] bfc0: b6ff5238 b6ff5690 00000000 00000078 be95ab6c 00000000 b6ff51d0 01e2b7c8
[  119.329337] bfe0: 00000078 be95aab0 b6f09253 b6eab746 20000030 01200011 00000000 00000000
[  119.337518] [<c0262500>] (obj_malloc) from [<c0262c4d>] (zs_malloc+0x149/0x364)
[  119.344838] [<c0262c4d>] (zs_malloc) from [<bf85ccd3>] (zram_bvec_rw.constprop.0+0x35b/0x52c [zram])
[  119.353979] [<bf85ccd3>] (zram_bvec_rw.constprop.0 [zram]) from [<bf85d22d>] (zram_rw_page+0xa1/0x120 [zram])
[  119.363890] [<bf85d22d>] (zram_rw_page [zram]) from [<c029f5a9>] (bdev_write_page+0x71/0x9c)
[  119.372329] [<c029f5a9>] (bdev_write_page) from [<c02477c7>] (__swap_writepage+0x47/0x280)
[  119.380591] [<c02477c7>] (__swap_writepage) from [<c02172b7>] (shrink_page_list+0x6e3/0xa74)
[  119.389022] [<c02172b7>] (shrink_page_list) from [<c0218217>] (shrink_inactive_list+0x15b/0x300)
[  119.397798] [<c0218217>] (shrink_inactive_list) from [<c0218983>] (shrink_lruvec+0x27b/0x444)
[  119.406313] [<c0218983>] (shrink_lruvec) from [<c0218c7d>] (shrink_node+0x131/0x4f0)
[  119.414049] [<c0218c7d>] (shrink_node) from [<c02190ff>] (do_try_to_free_pages+0xc3/0x2e0)
[  119.422305] [<c02190ff>] (do_try_to_free_pages) from [<c0219d11>] (try_to_free_pages+0xcd/0x1b4)
[  119.431083] [<c0219d11>] (try_to_free_pages) from [<c0243729>] (__alloc_pages_nodemask+0x3d9/0xc40)
[  119.440122] [<c0243729>] (__alloc_pages_nodemask) from [<c0230273>] (__pte_alloc+0x1f/0x118)
[  119.448552] [<c0230273>] (__pte_alloc) from [<c0230d7d>] (copy_page_range+0x455/0x554)
[  119.456467] [<c0230d7d>] (copy_page_range) from [<c011949b>] (dup_mm+0x21f/0x340)
[  119.463947] [<c011949b>] (dup_mm) from [<c011a759>] (copy_process+0xf75/0x1218)
[  119.471250] [<c011a759>] (copy_process) from [<c011ab5f>] (_do_fork+0x63/0x344)
[  119.478555] [<c011ab5f>] (_do_fork) from [<c011b07f>] (sys_clone+0x5b/0x74)
[  119.485514] [<c011b07f>] (sys_clone) from [<c0100061>] (ret_fast_syscall+0x1/0x62)
[  119.493071] Exception stack(0xc890bfa8 to 0xc890bff0)
[  119.498110] bfa0:                   b6ff5238 b6ff5690 01200011 00000000 00000000 00000000
[  119.506279] bfc0: b6ff5238 b6ff5690 00000000 00000078 be95ab6c 00000000 b6ff51d0 01e2b7c8
[  119.514444] bfe0: 00000078 be95aab0 b6f09253 b6eab746
[  119.519494] Code: 3629 2e00 dd08 2300 (6825) 3301
[  119.525481] ---[ end trace 0bcf63ee4f72ab80 ]---

 

The command run as root is:

:(){ :|: & };:

 

I noticed zs_malloc in the stack trace, and interestingly enough, it does not happen if I disable zram in /etc/default/armbian-zram-config. I get a flood of errors like below, and the device eventually recovers.

 

-bash: fork: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: Resource temporarily unavailable

 

I know the obvious answer is "don't do that" My memory requirements for this pi zero are modest, but stability is much more important. Can anyone suggest a good reason to not disable zram after seeing this? I also wonder where the right place to report this issue would be.

 

Thanks,

Alex

[Hammy]

I guess this is related to:

 

 

[Werner]

If so then it is possible that this change did not make it into the new release because there were stability issues with the 5.9 current branch that could not be fixed in time which is why the release has been built with older, known to work, 5.8 packages.

[Hammy]

Zram works for me with 5.9 build from trunk. I'm not sure though if building from trunk is a good idea when stability is a concern.

[atomic77]

Awesome, this does look related. I'm going to check a fresh image of Armbian 20.05 with stretch 5.4.45 kernel and confirm that this doesn't occur. Thanks!!

[atomic77]

Confirmed that I can't reproduce this on Stretch 5.4.45.

 

I'm leaning towards just disabling zram on 5.8 kernels - as far as I can tell, the main thing I need to watch out for is that I don't fill up 50MB of logs within the 15 minute interval of armbian-truncate-logs being run (now that it's not compressed)?