Sonic

Hello! I am experiencing the same issue on OrangePi +2e with: root@orangepiplus2e:/# uname -a Linux orangepiplus2e 4.19.62-sunxi #5.92 SMP Wed Jul 31 22:07:23 CEST 2019 armv7l GNU/Linux It sporadically crashes after about 3 - 5 weeks of running 24h. It is painful because it runs OpenHAB and controls my home. :( It feels like the kernel is OK but it cannot launch any process. Symptoms are: 1. Home automation stops working completely 2. nginx (responsible for externally visible web interface) still works, but returns error 500. 3. The board still responds to PINGs 4. Attempt to connect via ssh fails with "Connection refused" 5. Serial port responds, but only until you press "Enter". The last thing i see is "orangepi TTYs0" line, then silence. No login prompt. 6. SysRq l (show-backtrace-all-active-cpus) outputs nothing. Periodically it dumps: [75677.336667] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [75677.342628] rcu: 2-...!: (10 GPs behind) idle=988/0/0x0 softirq=5109306/5109306 fqs=0 [75677.350661] rcu: 3-...!: (1 GPs behind) idle=b3c/0/0x0 softirq=5733199/5733199 fqs=0 [75677.358594] rcu: (detected by 1, t=5252 jiffies, g=8503225, q=2) [75677.364715] Sending NMI from CPU 1 to CPUs 2: [75677.369403] NMI backtrace for cpu 2 [75677.369411] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.19.62-sunxi #5.92 [75677.369416] Hardware name: Allwinner sun8i Family [75677.369420] PC is at arch_cpu_idle+0x28/0x2c [75677.369424] LR is at arch_cpu_idle+0x27/0x2c [75677.369430] pc : [<c01078f4>] lr : [<c01078f3>] psr: 400f0133 [75677.369434] sp : ee553fb0 ip : 00000000 fp : 00000000 [75677.369439] r10: c0dba870 r9 : c0e04d48 r8 : 00000000 [75677.369445] r7 : 00000004 r6 : c0e04db8 r5 : c0e04d70 r4 : ffffe000 [75677.369450] r3 : c0116441 r2 : eedc2438 r1 : 03d93988 r0 : 00000000 [75677.369456] Flags: nZcv IRQs on FIQs on Mode SVC_32 ISA Thumb Segment none [75677.369461] Control: 50c5387d Table: 6c94c06a DAC: 00000051 [75677.369467] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.19.62-sunxi #5.92 [75677.369471] Hardware name: Allwinner sun8i Family [75677.369478] [<c010d74d>] (unwind_backtrace) from [<c010a2f1>] (show_stack+0x11/0x14) [75677.369484] [<c010a2f1>] (show_stack) from [<c08fc121>] (dump_stack+0x69/0x78) [75677.369490] [<c08fc121>] (dump_stack) from [<c090036d>] (nmi_cpu_backtrace+0x59/0x90) [75677.369497] [<c090036d>] (nmi_cpu_backtrace) from [<c010c5d9>] (handle_IPI+0x85/0x2c0) [75677.369503] [<c010c5d9>] (handle_IPI) from [<c05c9c7f>] (gic_handle_irq+0x67/0x68) [75677.369509] [<c05c9c7f>] (gic_handle_irq) from [<c0101a65>] (__irq_svc+0x65/0x94) [75677.369514] Exception stack(0xee553f60 to 0xee553fa8) [75677.369521] 3f60: 00000000 03d93988 eedc2438 c0116441 ffffe000 c0e04d70 c0e04db8 00000004 [75677.369527] 3f80: 00000000 c0e04d48 c0dba870 00000000 00000000 ee553fb0 c01078f3 c01078f4 [75677.369531] 3fa0: 400f0133 ffffffff [75677.369537] [<c0101a65>] (__irq_svc) from [<c01078f4>] (arch_cpu_idle+0x28/0x2c) [75677.369543] [<c01078f4>] (arch_cpu_idle) from [<c013e96b>] (do_idle+0x14b/0x1d8) [75677.369549] [<c013e96b>] (do_idle) from [<c013ebed>] (cpu_startup_entry+0x19/0x1c) [75677.369555] [<c013ebed>] (cpu_startup_entry) from [<401024b1>] (0x401024b1) [75677.370094] Sending NMI from CPU 1 to CPUs 3: [75677.552324] NMI backtrace for cpu 3 [75677.552331] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.19.62-sunxi #5.92 [75677.552336] Hardware name: Allwinner sun8i Family [75677.552340] PC is at arch_cpu_idle+0x28/0x2c [75677.552345] LR is at arch_cpu_idle+0x27/0x2c [75677.552350] pc : [<c01078f4>] lr : [<c01078f3>] psr: 40070033 [75677.552355] sp : ee555fb0 ip : 7fffffff fp : 00000000 [75677.552360] r10: c0dba870 r9 : c0e04d48 r8 : 00000000 [75677.552365] r7 : 00000008 r6 : c0e04db8 r5 : c0e04d70 r4 : ffffe000 [75677.552371] r3 : c0116441 r2 : eedd4438 r1 : 02b05b3c r0 : 00000000 [75677.552377] Flags: nZcv IRQs on FIQs on Mode SVC_32 ISA Thumb Segment none [75677.552382] Control: 50c5387d Table: 6c4e806a DAC: 00000051 [75677.552388] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.19.62-sunxi #5.92 [75677.552393] Hardware name: Allwinner sun8i Family [75677.552399] [<c010d74d>] (unwind_backtrace) from [<c010a2f1>] (show_stack+0x11/0x14) [75677.552405] [<c010a2f1>] (show_stack) from [<c08fc121>] (dump_stack+0x69/0x78) [75677.552411] [<c08fc121>] (dump_stack) from [<c090036d>] (nmi_cpu_backtrace+0x59/0x90) [75677.552418] [<c090036d>] (nmi_cpu_backtrace) from [<c010c5d9>] (handle_IPI+0x85/0x2c0) [75677.552424] [<c010c5d9>] (handle_IPI) from [<c05c9c7f>] (gic_handle_irq+0x67/0x68) [75677.552430] [<c05c9c7f>] (gic_handle_irq) from [<c0101a65>] (__irq_svc+0x65/0x94) [75677.552435] Exception stack(0xee555f60 to 0xee555fa8) [75677.552442] 5f60: 00000000 02b05b3c eedd4438 c0116441 ffffe000 c0e04d70 c0e04db8 00000008 [75677.552448] 5f80: 00000000 c0e04d48 c0dba870 00000000 7fffffff ee555fb0 c01078f3 c01078f4 [75677.552452] 5fa0: 40070033 ffffffff [75677.552458] [<c0101a65>] (__irq_svc) from [<c01078f4>] (arch_cpu_idle+0x28/0x2c) [75677.552464] [<c01078f4>] (arch_cpu_idle) from [<c013e96b>] (do_idle+0x14b/0x1d8) [75677.552470] [<c013e96b>] (do_idle) from [<c013ebed>] (cpu_startup_entry+0x19/0x1c) [75677.552476] [<c013ebed>] (cpu_startup_entry) from [<401024b1>] (0x401024b1) [75677.553015] rcu: rcu_sched kthread starved for 5305 jiffies! g8503225 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=1 [75677.741423] rcu: RCU grace-period kthread stack dump: [75677.746469] rcu_sched I 0 10 2 0x00000000 [75677.751961] [<c090acdb>] (__schedule) from [<c090b247>] (schedule+0x2f/0x68) [75677.759008] [<c090b247>] (schedule) from [<c090daf7>] (schedule_timeout+0x77/0x320) [75677.766667] [<c090daf7>] (schedule_timeout) from [<c016af6f>] (rcu_gp_kthread+0x41f/0x728) [75677.774933] [<c016af6f>] (rcu_gp_kthread) from [<c0132ae9>] (kthread+0xfd/0x104) [75677.782328] [<c0132ae9>] (kthread) from [<c01010f9>] (ret_from_fork+0x11/0x38) [75677.789545] Exception stack(0xee54bfb0 to 0xee54bff8) [75677.794593] bfa0: 00000000 00000000 00000000 00000000 [75677.802768] bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [75677.810942] bfe0: 00000000 00000000 00000000 00000000 00000013 00000000 I also have SysRq t ("Show task states") output, but it's way too huge to post here. Any recommendations ?