bprinz Posted July 24, 2017 Posted July 24, 2017 (edited) Hi, we have found an issue with Armbian 5.31 on a Banana Pi M2+ board that occurs when IP forwarding is enabled and there is high network load. How to reproduce this issue: Enable IP forwarding on the M2+: sysctl -w net.ipv4.ip_forward=1 Install nginx or another web server on the M2+ Use ncat to send a large file (e.g. 4 GB) from one host to another via the M2+ Use ab to create additional load on the nginx server Soon there will be a kernel panic, for example (see attachment for more examples): Reveal hidden contents [ 169.520266] Unable to handle kernel NULL pointer dereference at virtual address 00000004 [ 169.529459] pgd = c0004000 [ 169.530016] [00000004] *pgd=00000000 [ 169.530016] sunxi oops: enable sdcard JTAG interface [ 169.530016] sunxi oops: cpu frequency: 1200 MHz [ 169.530016] sunxi oops: ddr frequency: 672 MHz [ 169.530016] sunxi oops: gpu frequency: 105 MHz [ 169.530016] sunxi oops: cpu temperature: 47 [ 169.530016] Internal error: Oops: 817 [#1] PREEMPT SMP ARM [ 169.530016] Modules linked in: bnep ir_lirc_codec lirc_dev ir_mce_kbd_decoder ir_sanyo_decoder ir_sony_decoder ir_jvc_decoder ir_rc6_decoder ir_rc5_decoder ir_nec_decoder sunxi_cir rc_core pcf8591 bmp085 hidp rfcomm hci_uart bluetooth dhd btrfs [last unloaded: scsi_wait_scan] [ 169.530016] CPU: 0 Not tainted (3.4.113-sun8i #18) [ 169.530016] PC is at skb_dequeue+0x5c/0x74 [ 169.530016] LR is at _raw_spin_lock_irqsave+0x18/0x1c [ 169.530016] pc : [<c04f3e7c>] lr : [<c061da5c>] psr: a00f0093 [ 169.530016] sp : ef0a3e78 ip : 00000000 fp : ef0a3e94 [ 169.530016] r10: 00000040 r9 : c0970e48 r8 : d0b12bd0 [ 169.530016] r7 : 000000bd r6 : eea09538 r5 : eea0952c r4 : ee5d3900 [ 169.530016] r3 : eea0952c r2 : 00000000 r1 : 600f0013 r0 : 600f0013 [ 169.530016] Flags: NzCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment kernel [ 169.530016] Control: 10c5387d Table: 6e35406a DAC: 00000015 [ 169.530016] [ 169.530016] PC: 0xc04f3dfc: [ 169.530016] 3dfc e28320b4 e1921f9f e2811001 e1820f91 e3300000 1afffffa e5933000 eafffff5 [ 169.530016] 3e1c e89da800 e1a0c00d e92dd8f0 e24cb004 e52de004 e8bd4000 e280600c e1a05000 [ 169.530016] 3e3c e1a00006 eb04a6ff e5954000 e1550004 03a04000 e1a01000 0a00000a e3540000 [ 169.530016] 3e5c 0a000008 e5953008 e3a0c000 e2433001 e5853008 e894000c e584c004 e584c000 [ 169.530016] 3e7c e5823004 e5832000 e1a00006 eb04a529 e1a00004 e89da8f0 e1a0c00d e92dd8f0 [ 169.530016] 3e9c e24cb004 e52de004 e8bd4000 e280600c e1a05000 e1a00006 eb04a6e2 e5954004 [ 169.530016] 3ebc e1550004 03a04000 e1a01000 0a00000a e3540000 0a000008 e5953008 e3a0c000 [ 169.530016] 3edc e2433001 e5853008 e894000c e584c004 e584c000 e5823004 e5832000 e1a00006 [ 169.530016] [ 169.530016] LR: 0xc061d9dc: [ 169.530016] d9dc e3330000 01853f92 e3530000 0a000012 e121f000 e5943004 e2433001 e5843004 [ 169.530016] d9fc e5943000 e3130002 0a000000 ebfffbd7 e5953004 e3530000 03a03001 05853004 [ 169.530016] da1c e5953000 e3530000 0affffe5 e5953004 e3530000 0affffe2 eafffff8 f57ff05f [ 169.530016] da3c e5853004 e89da830 e1a0c00d e92dd800 e24cb004 e52de004 e8bd4000 ebffffcf [ 169.530016] da5c e89da800 e1a0c00d e92dd800 e24cb004 e52de004 e8bd4000 ebfffff2 e89da800 [ 169.530016] da7c e1a0c00d e92dd800 e24cb004 e52de004 e8bd4000 ebffffeb e89da800 e1a0c00d [ 169.530016] da9c e92dd830 e24cb004 e52de004 e8bd4000 ebffffba e1a04000 ebe84909 e121f004 [ 169.530016] dabc e89da830 e1a0c00d e92dd800 e24cb004 e52de004 e8bd4000 ebffffef e89da800 [ 169.530016] [ 169.530016] SP: 0xef0a3df8: [ 169.530016] 3df8 00000000 00000800 ef0a3e3c ef0a3e10 c0501a30 c00aa69c c04f3e7c a00f0093 [ 169.530016] 3e18 ffffffff ef0a3e64 ef0a3e94 ef0a3e30 c000dad8 c0008370 600f0013 600f0013 [ 169.530016] 3e38 00000000 eea0952c ee5d3900 eea0952c eea09538 000000bd d0b12bd0 c0970e48 [ 169.530016] 3e58 00000040 ef0a3e94 00000000 ef0a3e78 c061da5c c04f3e7c a00f0093 ffffffff [ 169.530016] 3e78 eea09000 c0970df0 c0a3e414 000000bd ef0a3ec4 ef0a3e98 c03ec8ec c04f3e2c [ 169.530016] 3e98 00000002 00000000 eea09548 eea09548 ee799140 c001a570 00000040 6a33da6c [ 169.530016] 3eb8 ef0a3f04 ef0a3ec8 c03eed94 c03ec868 00000000 c00aa234 c13ddaa8 6e325040 [ 169.530016] 3ed8 c09455e0 eea09548 c13dfe00 00000001 ef0a2000 000000ec 00000040 c098e469 [ 169.530016] [ 169.530016] FP: 0xef0a3e14: [ 169.530016] 3e14 a00f0093 ffffffff ef0a3e64 ef0a3e94 ef0a3e30 c000dad8 c0008370 600f0013 [ 169.530016] 3e34 600f0013 00000000 eea0952c ee5d3900 eea0952c eea09538 000000bd d0b12bd0 [ 169.530016] 3e54 c0970e48 00000040 ef0a3e94 00000000 ef0a3e78 c061da5c c04f3e7c a00f0093 [ 169.530016] 3e74 ffffffff eea09000 c0970df0 c0a3e414 000000bd ef0a3ec4 ef0a3e98 c03ec8ec [ 169.530016] 3e94 c04f3e2c 00000002 00000000 eea09548 eea09548 ee799140 c001a570 00000040 [ 169.530016] 3eb4 6a33da6c ef0a3f04 ef0a3ec8 c03eed94 c03ec868 00000000 c00aa234 c13ddaa8 [ 169.530016] 3ed4 6e325040 c09455e0 eea09548 c13dfe00 00000001 ef0a2000 000000ec 00000040 [ 169.530016] 3ef4 c098e469 ef0a3f44 ef0a3f08 c05028e0 c03eec04 00000001 c09240c0 c13dfe08 [ 169.530016] [ 169.530016] R3: 0xeea094ac: [ 169.530016] 94ac 00000000 00000000 00000000 00000000 00000000 ee0e5800 00000000 00000000 [ 169.530016] 94cc 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 169.530016] 94ec 00000000 00000000 00000000 00000000 00000000 d0b11000 ee054c00 000000ea [ 169.530016] 950c 000000eb 50b11000 000007ff d0b12000 ee148400 000000bd 000000d7 50b12000 [ 169.530016] 952c ee6ea200 ee6e7a80 00000019 00000001 00000000 eea09000 c0970e48 c13dfe08 [ 169.530016] 954c c13dfe08 00000001 00000040 c03eebf8 00000001 eea09000 eea09094 eea09094 [ 169.530016] 956c ee799140 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 169.530016] 958c 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 169.530016] [ 169.530016] R4: 0xee5d3880: [ 169.530016] 3880 463be812 00000000 00000000 00000000 00000000 00000000 0000000e 00000000 [ 169.530016] 38a0 00000000 00080006 c04efc18 00000000 00000000 00000000 00000002 00000000 [ 169.530016] 38c0 00000000 00000000 00000000 00000000 00000000 ee27b062 ee27b04e ee27b040 [ 169.530016] 38e0 ee27b082 ee27bf40 ee27b000 ee27b082 000010c0 00000001 00004372 00004373 [ 169.530016] 3900 00000000 ee6ea200 00000000 00000000 00000000 00000000 00000000 00000000 [ 169.530016] 3920 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 169.530016] 3940 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 169.530016] 3960 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 169.530016] [ 169.530016] R5: 0xeea094ac: [ 169.530016] 94ac 00000000 00000000 00000000 00000000 00000000 ee0e5800 00000000 00000000 [ 169.530016] 94cc 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 169.530016] 94ec 00000000 00000000 00000000 00000000 00000000 d0b11000 ee054c00 000000f6 [ 169.530016] 950c 000000f7 50b11000 000007ff d0b12000 ee148400 000000bd 000000d7 50b12000 [ 169.530016] 952c ee6ea200 ee6e7a80 00000019 00000001 00000000 eea09000 c0970e48 c13dfe08 [ 169.530016] 954c c13dfe08 00000001 00000040 c03eebf8 00000001 eea09000 eea09094 eea09094 [ 169.530016] 956c ee799140 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 169.530016] 958c 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 169.530016] [ 169.530016] R6: 0xeea094b8: [ 169.530016] 94b8 00000000 00000000 ee0e5800 00000000 00000000 00000000 00000000 00000000 [ 169.530016] 94d8 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 169.530016] 94f8 00000000 00000000 d0b11000 ee054c00 000000f6 000000f7 50b11000 000007ff [ 169.530016] 9518 d0b12000 ee148400 000000bd 000000d7 50b12000 ee6ea200 ee6e7a80 00000019 [ 169.530016] 9538 00000001 00000000 eea09000 c0970e48 c13dfe08 c13dfe08 00000001 00000040 [ 169.530016] 9558 c03eebf8 00000001 eea09000 eea09094 eea09094 ee799140 00000000 00000000 [ 169.530016] 9578 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 169.530016] 9598 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 169.530016] [ 169.530016] R8: 0xd0b12b50: [ 169.530016] 2b50 00520320 010007ff 6e1ef040 50b12b60 00520320 010007ff 6e273040 50b12b70 [ 169.530016] 2b70 00520320 010007ff 6e1f8040 50b12b80 00520320 010007ff 6e289040 50b12b90 [ 169.530016] 2b90 00520320 010007ff 6ea6a040 50b12ba0 00520320 010007ff 6e205040 50b12bb0 [ 169.530016] 2bb0 00520320 010007ff 6e298040 50b12bc0 00520320 010007ff 6e516040 50b12bd0 [ 169.530016] 2bd0 05ee0320 010007ff 6e6cd040 50b12be0 05ee0320 010007ff 6e7d8040 50b12bf0 [ 169.530016] 2bf0 05ee0320 010007ff 6e6ba040 50b12c00 05ee0320 010007ff 6e203040 50b12c10 [ 169.530016] 2c10 05ee0320 010007ff 6e1b5040 50b12c20 05ee0320 010007ff 6d8d0040 50b12c30 [ 169.530016] 2c30 05ee0320 010007ff 6e210040 50b12c40 05ee0320 010007ff 6e7c7040 50b12c50 [ 169.530016] [ 169.530016] R9: 0xc0970dc8: [ 169.530016] 0dc8 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 169.530016] 0de8 00000000 00000000 c03ee2f0 c03ed774 00000000 00000000 00000000 c07d91a3 [ 169.530016] 0e08 c096ef60 00000000 00000000 00000000 00000000 c03b9838 c03b987c 00000000 [ 169.530016] 0e28 00000000 00000000 00000000 c06be4f8 ee9fd5c0 00000000 c07d91a3 ffffffff [ 169.530016] 0e48 c096ede0 ee9fd640 eea0a980 eea092dc eea0620c c096ede8 ef0cfac0 c096ec78 [ 169.530016] 0e68 eea0bc88 0000000b 00000007 00000000 00000000 00000001 00000000 00000000 [ 169.530016] 0e88 c0970e88 c0970e88 00000000 c096ef60 c0970e04 00000000 00000000 00000000 [ 169.530016] 0ea8 00000000 00000000 eea09338 eea06268 7fffffff 00000000 00000000 c0970ec4 [ 169.530016] Process ksoftirqd/0 (pid: 3, stack limit = 0xef0a22f8) [ 169.530016] Stack: (0xef0a3e78 to 0xef0a4000) [ 169.530016] 3e60: eea09000 c0970df0 [ 169.530016] 3e80: c0a3e414 000000bd ef0a3ec4 ef0a3e98 c03ec8ec c04f3e2c 00000002 00000000 [ 169.530016] 3ea0: eea09548 eea09548 ee799140 c001a570 00000040 6a33da6c ef0a3f04 ef0a3ec8 [ 169.530016] 3ec0: c03eed94 c03ec868 00000000 c00aa234 c13ddaa8 6e325040 c09455e0 eea09548 [ 169.530016] 3ee0: c13dfe00 00000001 ef0a2000 000000ec 00000040 c098e469 ef0a3f44 ef0a3f08 [ 169.530016] 3f00: c05028e0 c03eec04 00000001 c09240c0 c13dfe08 ffffcd09 ef0a3f44 ef0a2000 [ 169.530016] 3f20: c092408c 00000003 00000001 c09b3980 00000101 00000009 ef0a3f8c ef0a3f48 [ 169.530016] 3f40: c0030a20 c050280c ef0a3f8c ef0a3f58 c00aba30 c09240c0 00000000 ffffcd08 [ 169.530016] 3f60: c0058c54 ef0a2000 00000000 00000000 00000001 c09b3980 c0620264 00000000 [ 169.530016] 3f80: ef0a3fb4 ef0a3f90 c0030c44 c0030948 ef037eec 00000000 c0030b7c 00000013 [ 169.530016] 3fa0: 00000000 00000000 ef0a3ff4 ef0a3fb8 c0049cf0 c0030b88 00000000 00000000 [ 169.530016] 3fc0: 00000000 00000000 00000000 00000000 ef0a3fd0 ef0a3fd0 00000000 ef037eec [ 169.530016] 3fe0: c0049c5c c000f1b0 00000000 ef0a3ff8 c000f1b0 c0049c68 ffffffff ffffffff [ 169.530016] [<c04f3e7c>] (skb_dequeue+0x5c/0x74) from [<c03ec8ec>] (geth_rx_refill+0x90/0x180) [ 169.530016] [<c03ec8ec>] (geth_rx_refill+0x90/0x180) from [<c03eed94>] (geth_poll+0x19c/0x1cc) [ 170.983490] [<c03eed94>] (geth_poll+0x19c/0x1cc) from [<c05028e0>] (net_rx_action+0xe0/0x1e0) [ 170.983490] [<c05028e0>] (net_rx_action+0xe0/0x1e0) from [<c0030a20>] (__do_softirq+0xe4/0x240) [ 170.983490] [<c0030a20>] (__do_softirq+0xe4/0x240) from [<c0030c44>] (run_ksoftirqd+0xc8/0x190) [ 170.983490] [<c0030c44>] (run_ksoftirqd+0xc8/0x190) from [<c0049cf0>] (kthread+0x94/0xa0) [ 170.983490] [<c0049cf0>] (kthread+0x94/0xa0) from [<c000f1b0>] (kernel_thread_exit+0x0/0x8) [ 170.983490] Code: e5853008 e894000c e584c004 e584c000 (e5823004) [ 171.068055] ---[ end trace ec6e0286501d0856 ]--- [ 171.077701] Kernel panic - not syncing: Fatal exception in interrupt [ 171.089158] CPU1: stopping [ 171.096578] [<c0016a20>] (unwind_backtrace+0x0/0xe8) from [<c0615d20>] (dump_stack+0x20/0x24) [ 171.099154] [<c0615d20>] (dump_stack+0x20/0x24) from [<c0014170>] (handle_IPI+0x114/0x298) [ 171.099154] [<c0014170>] (handle_IPI+0x114/0x298) from [<c0008554>] (gic_handle_irq+0x64/0x6c) [ 171.099154] [<c0008554>] (gic_handle_irq+0x64/0x6c) from [<c000db40>] (__irq_svc+0x40/0x74) [ 171.099154] Exception stack(0xef0bdf68 to 0xef0bdfb0) [ 171.099154] df60: c13e5b38 c094430c 0000000f 00000000 ef0bc000 00000001 [ 171.099154] df80: ef0bc000 c0992964 4000406a 410fc075 00000000 ef0bdfbc ef0bdfc0 ef0bdfb0 [ 171.099154] dfa0: c000f2d8 c000f2dc 60000013 ffffffff [ 171.099154] [<c000db40>] (__irq_svc+0x40/0x74) from [<c000f2dc>] (default_idle+0x34/0x3c) [ 171.099154] [<c000f2dc>] (default_idle+0x34/0x3c) from [<c000f694>] (cpu_idle+0xa8/0xf4) [ 171.099154] [<c000f694>] (cpu_idle+0xa8/0xf4) from [<c0612c1c>] (secondary_start_kernel+0x108/0x12c) [ 171.099154] [<c0612c1c>] (secondary_start_kernel+0x108/0x12c) from [<40612234>] (0x40612234) [ 171.099154] CPU3: stopping [ 171.099154] [<c0016a20>] (unwind_backtrace+0x0/0xe8) from [<c0615d20>] (dump_stack+0x20/0x24) [ 171.099154] [<c0615d20>] (dump_stack+0x20/0x24) from [<c0014170>] (handle_IPI+0x114/0x298) [ 171.099154] [<c0014170>] (handle_IPI+0x114/0x298) from [<c0008554>] (gic_handle_irq+0x64/0x6c) [ 171.099154] [<c0008554>] (gic_handle_irq+0x64/0x6c) from [<c000db40>] (__irq_svc+0x40/0x74) [ 171.099154] Exception stack(0xef0fbf68 to 0xef0fbfb0) [ 171.099154] bf60: c13f5b38 c094430c 0000000f 00000000 ef0fa000 00000003 [ 171.099154] bf80: ef0fa000 c0992964 4000406a 410fc075 00000000 ef0fbfbc ef0fbfc0 ef0fbfb0 [ 171.099154] bfa0: c000f2d8 c000f2dc 600d0013 ffffffff [ 171.099154] [<c000db40>] (__irq_svc+0x40/0x74) from [<c000f2dc>] (default_idle+0x34/0x3c) [ 171.099154] [<c000f2dc>] (default_idle+0x34/0x3c) from [<c000f694>] (cpu_idle+0xa8/0xf4) [ 171.099154] [<c000f694>] (cpu_idle+0xa8/0xf4) from [<c0612c1c>] (secondary_start_kernel+0x108/0x12c) [ 171.099154] [<c0612c1c>] (secondary_start_kernel+0x108/0x12c) from [<40612234>] (0x40612234) [ 171.099154] CPU2: stopping [ 171.099154] [<c0016a20>] (unwind_backtrace+0x0/0xe8) from [<c0615d20>] (dump_stack+0x20/0x24) [ 171.099154] [<c0615d20>] (dump_stack+0x20/0x24) from [<c0014170>] (handle_IPI+0x114/0x298) [ 171.099154] [<c0014170>] (handle_IPI+0x114/0x298) from [<c0008554>] (gic_handle_irq+0x64/0x6c) [ 171.099154] [<c0008554>] (gic_handle_irq+0x64/0x6c) from [<c000db40>] (__irq_svc+0x40/0x74) [ 171.099154] Exception stack(0xef0edf68 to 0xef0edfb0) [ 171.099154] df60: c13edb38 c094430c 0000000f 00000000 ef0ec000 00000002 [ 171.099154] df80: ef0ec000 c0992964 4000406a 410fc075 00000000 ef0edfbc ef0edfc0 ef0edfb0 [ 171.099154] dfa0: c000f2d8 c000f2dc 60000013 ffffffff [ 171.099154] [<c000db40>] (__irq_svc+0x40/0x74) from [<c000f2dc>] (default_idle+0x34/0x3c) [ 171.099154] [<c000f2dc>] (default_idle+0x34/0x3c) from [<c000f694>] (cpu_idle+0xa8/0xf4) [ 171.099154] [<c000f694>] (cpu_idle+0xa8/0xf4) from [<c0612c1c>] (secondary_start_kernel+0x108/0x12c) [ 171.099154] [<c0612c1c>] (secondary_start_kernel+0x108/0x12c) from [<40612234>] (0x40612234) [ 171.087659] [hotplug]: cpu(0) try to kill cpu(1) [ 171.087659] [hotplug]: try to kill cpu:1 failed! [ 171.087659] [hotplug]: cpu(0) try to kill cpu(2) [ 171.087659] [hotplug]: try to kill cpu:2 failed! [ 171.087659] [hotplug]: cpu(0) try to kill cpu(3) [ 171.087659] [hotplug]: try to kill cpu:3 failed! [ 171.087659] Rebooting in 10 seconds.. [ 171.087659] Restarting Linux version 3.4.113-sun8i (root@devel) (gcc version 5.4.0 20160609 (Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.4) ) #18 SMP PREEMPT Thu Jun 15 02:16:06 CEST 2017 [ 171.087659] Before the kernel panic the ksoftirqd thread uses 100% of one core. Best regards, bprinz logs.zipFetching info... Edited July 24, 2017 by Tido added spoiler
tkaiser Posted July 24, 2017 Posted July 24, 2017 On 7/24/2017 at 10:17 AM, bprinz said: ddr frequency: 672 MHz Expand This is wrong. Fex file still contains 624 MHz so upstream u-boot uses moronically 672 MHz here and our patch to fix this has vanished
bprinz Posted July 24, 2017 Author Posted July 24, 2017 On 7/24/2017 at 11:46 AM, adrb said: Did you try disabling bnep module? Expand I have now blacklisted the bnep module: ksoftirqd uses less CPU, IP forwarding is a bit slower, but the kernel panic still occurs after a while.
adrb Posted July 24, 2017 Posted July 24, 2017 On 7/24/2017 at 2:09 PM, bprinz said: but the kernel panic still occurs after a while. Expand Corrupted sk_buf list means that it's either buggy driver or memory corruption. Lower your ddr frequency as tkaiser sugests.
tkaiser Posted July 24, 2017 Posted July 24, 2017 On 7/24/2017 at 3:13 PM, adrb said: Lower your ddr frequency as tkaiser sugests Expand sudo h3consumption -d 624 && sudo reboot
bprinz Posted July 24, 2017 Author Posted July 24, 2017 Thank you for the hint. At 624 MHz I still get the kernel panic, with and without bnep. kernel-panic-11-624MHz-no_bnep.txt
tkaiser Posted July 24, 2017 Posted July 24, 2017 On 7/24/2017 at 3:38 PM, bprinz said: At 624 MHz I still get the kernel panic, with and without bnep. Expand Well, at least for me all Banana Pi are in the meantime unsupported devices due to vendor's ignorance/stupidity. Sorry, will never again waste my time with Bananas (and especially not this crappy device that annihilated already so much of my time)
Recommended Posts