Jump to content

M2+: Kernel panic on high network load (3.4.113-sun8i)


bprinz

Recommended Posts

Hi,

 

we have found an issue with Armbian 5.31 on a Banana Pi M2+ board that occurs when IP forwarding is enabled and there is high network load.

 

How to reproduce this issue:

  1. Enable IP forwarding on the M2+: sysctl -w net.ipv4.ip_forward=1
  2. Install nginx or another web server on the M2+
  3. Use ncat to send a large file (e.g. 4 GB) from one host to another via the M2+
  4. Use ab to create additional load on the nginx server

 

Soon there will be a kernel panic, for example (see attachment for more examples):

Spoiler

[  169.520266] Unable to handle kernel NULL pointer dereference at virtual address 00000004
[  169.529459] pgd = c0004000
[  169.530016] [00000004] *pgd=00000000
[  169.530016] sunxi oops: enable sdcard JTAG interface
[  169.530016] sunxi oops: cpu frequency: 1200 MHz
[  169.530016] sunxi oops: ddr frequency: 672 MHz
[  169.530016] sunxi oops: gpu frequency: 105 MHz
[  169.530016] sunxi oops: cpu temperature: 47 
[  169.530016] Internal error: Oops: 817 [#1] PREEMPT SMP ARM
[  169.530016] Modules linked in: bnep ir_lirc_codec lirc_dev ir_mce_kbd_decoder ir_sanyo_decoder ir_sony_decoder ir_jvc_decoder ir_rc6_decoder ir_rc5_decoder ir_nec_decoder sunxi_cir rc_core pcf8591 bmp085 hidp rfcomm hci_uart bluetooth dhd btrfs [last unloaded: scsi_wait_scan]
[  169.530016] CPU: 0    Not tainted  (3.4.113-sun8i #18)
[  169.530016] PC is at skb_dequeue+0x5c/0x74
[  169.530016] LR is at _raw_spin_lock_irqsave+0x18/0x1c
[  169.530016] pc : [<c04f3e7c>]    lr : [<c061da5c>]    psr: a00f0093
[  169.530016] sp : ef0a3e78  ip : 00000000  fp : ef0a3e94
[  169.530016] r10: 00000040  r9 : c0970e48  r8 : d0b12bd0
[  169.530016] r7 : 000000bd  r6 : eea09538  r5 : eea0952c  r4 : ee5d3900
[  169.530016] r3 : eea0952c  r2 : 00000000  r1 : 600f0013  r0 : 600f0013
[  169.530016] Flags: NzCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment kernel
[  169.530016] Control: 10c5387d  Table: 6e35406a  DAC: 00000015
[  169.530016] 
[  169.530016] PC: 0xc04f3dfc:
[  169.530016] 3dfc  e28320b4 e1921f9f e2811001 e1820f91 e3300000 1afffffa e5933000 eafffff5
[  169.530016] 3e1c  e89da800 e1a0c00d e92dd8f0 e24cb004 e52de004 e8bd4000 e280600c e1a05000
[  169.530016] 3e3c  e1a00006 eb04a6ff e5954000 e1550004 03a04000 e1a01000 0a00000a e3540000
[  169.530016] 3e5c  0a000008 e5953008 e3a0c000 e2433001 e5853008 e894000c e584c004 e584c000
[  169.530016] 3e7c  e5823004 e5832000 e1a00006 eb04a529 e1a00004 e89da8f0 e1a0c00d e92dd8f0
[  169.530016] 3e9c  e24cb004 e52de004 e8bd4000 e280600c e1a05000 e1a00006 eb04a6e2 e5954004
[  169.530016] 3ebc  e1550004 03a04000 e1a01000 0a00000a e3540000 0a000008 e5953008 e3a0c000
[  169.530016] 3edc  e2433001 e5853008 e894000c e584c004 e584c000 e5823004 e5832000 e1a00006
[  169.530016] 
[  169.530016] LR: 0xc061d9dc:
[  169.530016] d9dc  e3330000 01853f92 e3530000 0a000012 e121f000 e5943004 e2433001 e5843004
[  169.530016] d9fc  e5943000 e3130002 0a000000 ebfffbd7 e5953004 e3530000 03a03001 05853004
[  169.530016] da1c  e5953000 e3530000 0affffe5 e5953004 e3530000 0affffe2 eafffff8 f57ff05f
[  169.530016] da3c  e5853004 e89da830 e1a0c00d e92dd800 e24cb004 e52de004 e8bd4000 ebffffcf
[  169.530016] da5c  e89da800 e1a0c00d e92dd800 e24cb004 e52de004 e8bd4000 ebfffff2 e89da800
[  169.530016] da7c  e1a0c00d e92dd800 e24cb004 e52de004 e8bd4000 ebffffeb e89da800 e1a0c00d
[  169.530016] da9c  e92dd830 e24cb004 e52de004 e8bd4000 ebffffba e1a04000 ebe84909 e121f004
[  169.530016] dabc  e89da830 e1a0c00d e92dd800 e24cb004 e52de004 e8bd4000 ebffffef e89da800
[  169.530016] 
[  169.530016] SP: 0xef0a3df8:
[  169.530016] 3df8  00000000 00000800 ef0a3e3c ef0a3e10 c0501a30 c00aa69c c04f3e7c a00f0093
[  169.530016] 3e18  ffffffff ef0a3e64 ef0a3e94 ef0a3e30 c000dad8 c0008370 600f0013 600f0013
[  169.530016] 3e38  00000000 eea0952c ee5d3900 eea0952c eea09538 000000bd d0b12bd0 c0970e48
[  169.530016] 3e58  00000040 ef0a3e94 00000000 ef0a3e78 c061da5c c04f3e7c a00f0093 ffffffff
[  169.530016] 3e78  eea09000 c0970df0 c0a3e414 000000bd ef0a3ec4 ef0a3e98 c03ec8ec c04f3e2c
[  169.530016] 3e98  00000002 00000000 eea09548 eea09548 ee799140 c001a570 00000040 6a33da6c
[  169.530016] 3eb8  ef0a3f04 ef0a3ec8 c03eed94 c03ec868 00000000 c00aa234 c13ddaa8 6e325040
[  169.530016] 3ed8  c09455e0 eea09548 c13dfe00 00000001 ef0a2000 000000ec 00000040 c098e469
[  169.530016] 
[  169.530016] FP: 0xef0a3e14:
[  169.530016] 3e14  a00f0093 ffffffff ef0a3e64 ef0a3e94 ef0a3e30 c000dad8 c0008370 600f0013
[  169.530016] 3e34  600f0013 00000000 eea0952c ee5d3900 eea0952c eea09538 000000bd d0b12bd0
[  169.530016] 3e54  c0970e48 00000040 ef0a3e94 00000000 ef0a3e78 c061da5c c04f3e7c a00f0093
[  169.530016] 3e74  ffffffff eea09000 c0970df0 c0a3e414 000000bd ef0a3ec4 ef0a3e98 c03ec8ec
[  169.530016] 3e94  c04f3e2c 00000002 00000000 eea09548 eea09548 ee799140 c001a570 00000040
[  169.530016] 3eb4  6a33da6c ef0a3f04 ef0a3ec8 c03eed94 c03ec868 00000000 c00aa234 c13ddaa8
[  169.530016] 3ed4  6e325040 c09455e0 eea09548 c13dfe00 00000001 ef0a2000 000000ec 00000040
[  169.530016] 3ef4  c098e469 ef0a3f44 ef0a3f08 c05028e0 c03eec04 00000001 c09240c0 c13dfe08
[  169.530016] 
[  169.530016] R3: 0xeea094ac:
[  169.530016] 94ac  00000000 00000000 00000000 00000000 00000000 ee0e5800 00000000 00000000
[  169.530016] 94cc  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  169.530016] 94ec  00000000 00000000 00000000 00000000 00000000 d0b11000 ee054c00 000000ea
[  169.530016] 950c  000000eb 50b11000 000007ff d0b12000 ee148400 000000bd 000000d7 50b12000
[  169.530016] 952c  ee6ea200 ee6e7a80 00000019 00000001 00000000 eea09000 c0970e48 c13dfe08
[  169.530016] 954c  c13dfe08 00000001 00000040 c03eebf8 00000001 eea09000 eea09094 eea09094
[  169.530016] 956c  ee799140 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  169.530016] 958c  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  169.530016] 
[  169.530016] R4: 0xee5d3880:
[  169.530016] 3880  463be812 00000000 00000000 00000000 00000000 00000000 0000000e 00000000
[  169.530016] 38a0  00000000 00080006 c04efc18 00000000 00000000 00000000 00000002 00000000
[  169.530016] 38c0  00000000 00000000 00000000 00000000 00000000 ee27b062 ee27b04e ee27b040
[  169.530016] 38e0  ee27b082 ee27bf40 ee27b000 ee27b082 000010c0 00000001 00004372 00004373
[  169.530016] 3900  00000000 ee6ea200 00000000 00000000 00000000 00000000 00000000 00000000
[  169.530016] 3920  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  169.530016] 3940  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  169.530016] 3960  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  169.530016] 
[  169.530016] R5: 0xeea094ac:
[  169.530016] 94ac  00000000 00000000 00000000 00000000 00000000 ee0e5800 00000000 00000000
[  169.530016] 94cc  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  169.530016] 94ec  00000000 00000000 00000000 00000000 00000000 d0b11000 ee054c00 000000f6
[  169.530016] 950c  000000f7 50b11000 000007ff d0b12000 ee148400 000000bd 000000d7 50b12000
[  169.530016] 952c  ee6ea200 ee6e7a80 00000019 00000001 00000000 eea09000 c0970e48 c13dfe08
[  169.530016] 954c  c13dfe08 00000001 00000040 c03eebf8 00000001 eea09000 eea09094 eea09094
[  169.530016] 956c  ee799140 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  169.530016] 958c  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  169.530016] 
[  169.530016] R6: 0xeea094b8:
[  169.530016] 94b8  00000000 00000000 ee0e5800 00000000 00000000 00000000 00000000 00000000
[  169.530016] 94d8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  169.530016] 94f8  00000000 00000000 d0b11000 ee054c00 000000f6 000000f7 50b11000 000007ff
[  169.530016] 9518  d0b12000 ee148400 000000bd 000000d7 50b12000 ee6ea200 ee6e7a80 00000019
[  169.530016] 9538  00000001 00000000 eea09000 c0970e48 c13dfe08 c13dfe08 00000001 00000040
[  169.530016] 9558  c03eebf8 00000001 eea09000 eea09094 eea09094 ee799140 00000000 00000000
[  169.530016] 9578  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  169.530016] 9598  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  169.530016] 
[  169.530016] R8: 0xd0b12b50:
[  169.530016] 2b50  00520320 010007ff 6e1ef040 50b12b60 00520320 010007ff 6e273040 50b12b70
[  169.530016] 2b70  00520320 010007ff 6e1f8040 50b12b80 00520320 010007ff 6e289040 50b12b90
[  169.530016] 2b90  00520320 010007ff 6ea6a040 50b12ba0 00520320 010007ff 6e205040 50b12bb0
[  169.530016] 2bb0  00520320 010007ff 6e298040 50b12bc0 00520320 010007ff 6e516040 50b12bd0
[  169.530016] 2bd0  05ee0320 010007ff 6e6cd040 50b12be0 05ee0320 010007ff 6e7d8040 50b12bf0
[  169.530016] 2bf0  05ee0320 010007ff 6e6ba040 50b12c00 05ee0320 010007ff 6e203040 50b12c10
[  169.530016] 2c10  05ee0320 010007ff 6e1b5040 50b12c20 05ee0320 010007ff 6d8d0040 50b12c30
[  169.530016] 2c30  05ee0320 010007ff 6e210040 50b12c40 05ee0320 010007ff 6e7c7040 50b12c50
[  169.530016] 
[  169.530016] R9: 0xc0970dc8:
[  169.530016] 0dc8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  169.530016] 0de8  00000000 00000000 c03ee2f0 c03ed774 00000000 00000000 00000000 c07d91a3
[  169.530016] 0e08  c096ef60 00000000 00000000 00000000 00000000 c03b9838 c03b987c 00000000
[  169.530016] 0e28  00000000 00000000 00000000 c06be4f8 ee9fd5c0 00000000 c07d91a3 ffffffff
[  169.530016] 0e48  c096ede0 ee9fd640 eea0a980 eea092dc eea0620c c096ede8 ef0cfac0 c096ec78
[  169.530016] 0e68  eea0bc88 0000000b 00000007 00000000 00000000 00000001 00000000 00000000
[  169.530016] 0e88  c0970e88 c0970e88 00000000 c096ef60 c0970e04 00000000 00000000 00000000
[  169.530016] 0ea8  00000000 00000000 eea09338 eea06268 7fffffff 00000000 00000000 c0970ec4
[  169.530016] Process ksoftirqd/0 (pid: 3, stack limit = 0xef0a22f8)
[  169.530016] Stack: (0xef0a3e78 to 0xef0a4000)
[  169.530016] 3e60:                                                       eea09000 c0970df0
[  169.530016] 3e80: c0a3e414 000000bd ef0a3ec4 ef0a3e98 c03ec8ec c04f3e2c 00000002 00000000
[  169.530016] 3ea0: eea09548 eea09548 ee799140 c001a570 00000040 6a33da6c ef0a3f04 ef0a3ec8
[  169.530016] 3ec0: c03eed94 c03ec868 00000000 c00aa234 c13ddaa8 6e325040 c09455e0 eea09548
[  169.530016] 3ee0: c13dfe00 00000001 ef0a2000 000000ec 00000040 c098e469 ef0a3f44 ef0a3f08
[  169.530016] 3f00: c05028e0 c03eec04 00000001 c09240c0 c13dfe08 ffffcd09 ef0a3f44 ef0a2000
[  169.530016] 3f20: c092408c 00000003 00000001 c09b3980 00000101 00000009 ef0a3f8c ef0a3f48
[  169.530016] 3f40: c0030a20 c050280c ef0a3f8c ef0a3f58 c00aba30 c09240c0 00000000 ffffcd08
[  169.530016] 3f60: c0058c54 ef0a2000 00000000 00000000 00000001 c09b3980 c0620264 00000000
[  169.530016] 3f80: ef0a3fb4 ef0a3f90 c0030c44 c0030948 ef037eec 00000000 c0030b7c 00000013
[  169.530016] 3fa0: 00000000 00000000 ef0a3ff4 ef0a3fb8 c0049cf0 c0030b88 00000000 00000000
[  169.530016] 3fc0: 00000000 00000000 00000000 00000000 ef0a3fd0 ef0a3fd0 00000000 ef037eec
[  169.530016] 3fe0: c0049c5c c000f1b0 00000000 ef0a3ff8 c000f1b0 c0049c68 ffffffff ffffffff
[  169.530016] [<c04f3e7c>] (skb_dequeue+0x5c/0x74) from [<c03ec8ec>] (geth_rx_refill+0x90/0x180)
[  169.530016] [<c03ec8ec>] (geth_rx_refill+0x90/0x180) from [<c03eed94>] (geth_poll+0x19c/0x1cc)
[  170.983490] [<c03eed94>] (geth_poll+0x19c/0x1cc) from [<c05028e0>] (net_rx_action+0xe0/0x1e0)
[  170.983490] [<c05028e0>] (net_rx_action+0xe0/0x1e0) from [<c0030a20>] (__do_softirq+0xe4/0x240)
[  170.983490] [<c0030a20>] (__do_softirq+0xe4/0x240) from [<c0030c44>] (run_ksoftirqd+0xc8/0x190)
[  170.983490] [<c0030c44>] (run_ksoftirqd+0xc8/0x190) from [<c0049cf0>] (kthread+0x94/0xa0)
[  170.983490] [<c0049cf0>] (kthread+0x94/0xa0) from [<c000f1b0>] (kernel_thread_exit+0x0/0x8)
[  170.983490] Code: e5853008 e894000c e584c004 e584c000 (e5823004) 
[  171.068055] ---[ end trace ec6e0286501d0856 ]---
[  171.077701] Kernel panic - not syncing: Fatal exception in interrupt
[  171.089158] CPU1: stopping
[  171.096578] [<c0016a20>] (unwind_backtrace+0x0/0xe8) from [<c0615d20>] (dump_stack+0x20/0x24)
[  171.099154] [<c0615d20>] (dump_stack+0x20/0x24) from [<c0014170>] (handle_IPI+0x114/0x298)
[  171.099154] [<c0014170>] (handle_IPI+0x114/0x298) from [<c0008554>] (gic_handle_irq+0x64/0x6c)
[  171.099154] [<c0008554>] (gic_handle_irq+0x64/0x6c) from [<c000db40>] (__irq_svc+0x40/0x74)
[  171.099154] Exception stack(0xef0bdf68 to 0xef0bdfb0)
[  171.099154] df60:                   c13e5b38 c094430c 0000000f 00000000 ef0bc000 00000001
[  171.099154] df80: ef0bc000 c0992964 4000406a 410fc075 00000000 ef0bdfbc ef0bdfc0 ef0bdfb0
[  171.099154] dfa0: c000f2d8 c000f2dc 60000013 ffffffff
[  171.099154] [<c000db40>] (__irq_svc+0x40/0x74) from [<c000f2dc>] (default_idle+0x34/0x3c)
[  171.099154] [<c000f2dc>] (default_idle+0x34/0x3c) from [<c000f694>] (cpu_idle+0xa8/0xf4)
[  171.099154] [<c000f694>] (cpu_idle+0xa8/0xf4) from [<c0612c1c>] (secondary_start_kernel+0x108/0x12c)
[  171.099154] [<c0612c1c>] (secondary_start_kernel+0x108/0x12c) from [<40612234>] (0x40612234)
[  171.099154] CPU3: stopping
[  171.099154] [<c0016a20>] (unwind_backtrace+0x0/0xe8) from [<c0615d20>] (dump_stack+0x20/0x24)
[  171.099154] [<c0615d20>] (dump_stack+0x20/0x24) from [<c0014170>] (handle_IPI+0x114/0x298)
[  171.099154] [<c0014170>] (handle_IPI+0x114/0x298) from [<c0008554>] (gic_handle_irq+0x64/0x6c)
[  171.099154] [<c0008554>] (gic_handle_irq+0x64/0x6c) from [<c000db40>] (__irq_svc+0x40/0x74)
[  171.099154] Exception stack(0xef0fbf68 to 0xef0fbfb0)
[  171.099154] bf60:                   c13f5b38 c094430c 0000000f 00000000 ef0fa000 00000003
[  171.099154] bf80: ef0fa000 c0992964 4000406a 410fc075 00000000 ef0fbfbc ef0fbfc0 ef0fbfb0
[  171.099154] bfa0: c000f2d8 c000f2dc 600d0013 ffffffff
[  171.099154] [<c000db40>] (__irq_svc+0x40/0x74) from [<c000f2dc>] (default_idle+0x34/0x3c)
[  171.099154] [<c000f2dc>] (default_idle+0x34/0x3c) from [<c000f694>] (cpu_idle+0xa8/0xf4)
[  171.099154] [<c000f694>] (cpu_idle+0xa8/0xf4) from [<c0612c1c>] (secondary_start_kernel+0x108/0x12c)
[  171.099154] [<c0612c1c>] (secondary_start_kernel+0x108/0x12c) from [<40612234>] (0x40612234)
[  171.099154] CPU2: stopping
[  171.099154] [<c0016a20>] (unwind_backtrace+0x0/0xe8) from [<c0615d20>] (dump_stack+0x20/0x24)
[  171.099154] [<c0615d20>] (dump_stack+0x20/0x24) from [<c0014170>] (handle_IPI+0x114/0x298)
[  171.099154] [<c0014170>] (handle_IPI+0x114/0x298) from [<c0008554>] (gic_handle_irq+0x64/0x6c)
[  171.099154] [<c0008554>] (gic_handle_irq+0x64/0x6c) from [<c000db40>] (__irq_svc+0x40/0x74)
[  171.099154] Exception stack(0xef0edf68 to 0xef0edfb0)
[  171.099154] df60:                   c13edb38 c094430c 0000000f 00000000 ef0ec000 00000002
[  171.099154] df80: ef0ec000 c0992964 4000406a 410fc075 00000000 ef0edfbc ef0edfc0 ef0edfb0
[  171.099154] dfa0: c000f2d8 c000f2dc 60000013 ffffffff
[  171.099154] [<c000db40>] (__irq_svc+0x40/0x74) from [<c000f2dc>] (default_idle+0x34/0x3c)
[  171.099154] [<c000f2dc>] (default_idle+0x34/0x3c) from [<c000f694>] (cpu_idle+0xa8/0xf4)
[  171.099154] [<c000f694>] (cpu_idle+0xa8/0xf4) from [<c0612c1c>] (secondary_start_kernel+0x108/0x12c)
[  171.099154] [<c0612c1c>] (secondary_start_kernel+0x108/0x12c) from [<40612234>] (0x40612234)
[  171.087659] [hotplug]: cpu(0) try to kill cpu(1)
[  171.087659] [hotplug]: try to kill cpu:1 failed!
[  171.087659] [hotplug]: cpu(0) try to kill cpu(2)
[  171.087659] [hotplug]: try to kill cpu:2 failed!
[  171.087659] [hotplug]: cpu(0) try to kill cpu(3)
[  171.087659] [hotplug]: try to kill cpu:3 failed!
[  171.087659] Rebooting in 10 seconds..
[  171.087659] Restarting Linux version 3.4.113-sun8i (root@devel) (gcc version 5.4.0 20160609 (Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.4) ) #18 SMP PREEMPT Thu Jun 15 02:16:06 CEST 2017
[  171.087659] 

 

Before the kernel panic the ksoftirqd thread uses 100% of one core.

 

Best regards,

bprinz

 

logs.zip

Edited by Tido
added spoiler
Link to comment
Share on other sites

2 hours ago, adrb said:

Did you try disabling bnep module?

 

I have now blacklisted the bnep module:

ksoftirqd uses less CPU, IP forwarding is a bit slower, but the kernel panic still occurs after a while.

Link to comment
Share on other sites

1 hour ago, bprinz said:

At 624 MHz I still get the kernel panic, with and without bnep.

 

Well, at least for me all Banana Pi are in the meantime unsupported devices due to vendor's ignorance/stupidity. Sorry, will never again waste my time with Bananas (and especially not this crappy device that annihilated already so much of my time)

Link to comment
Share on other sites

Guest
This topic is now closed to further replies.
×
×
  • Create New...

Important Information

Terms of Use - Privacy Policy - Guidelines