Mikhail Kulinich Posted August 13, 2019 Share Posted August 13, 2019 (edited) At 19:54:55 kernel crash happens. That for some reason forces system to change time to Aug 24 1978. systemd reacts to that situation strangely, which puts system under unresponsive state, like infinite loop, potentially causing 100% cpu occupation (this is guess now) The printouts "Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[20635]: Time has been changed" follows many times per second until node is rebooted manually. The kernel version: 4.19.50-sunxi #5.89 SMP Fri Jun 14 01:50:58 EDT 2019 armv7l armv7l armv7l GNU/Linux Aug 10 19:45:01 ptc_lab_collab3_node17 CRON[32019]: pam_unix(cron:session): session closed for user root Aug 10 19:45:01 ptc_lab_collab3_node17 CRON[32018]: pam_unix(cron:session): session closed for user root Aug 10 19:54:55 ptc_lab_collab3_node17 kernel: rcu: INFO: rcu_sched detected stalls on CPUs/tasks: Aug 10 19:54:55 ptc_lab_collab3_node17 kernel: rcu: 1-...!: (2 GPs behind) idle=bfa/0/0x1 softirq=4388853/4388853 fqs=0 Aug 10 19:54:55 ptc_lab_collab3_node17 kernel: rcu: 2-...!: (8 GPs behind) idle=114/0/0x0 softirq=4692641/4692641 fqs=0 Aug 10 19:54:55 ptc_lab_collab3_node17 kernel: rcu: (detected by 3, t=39728 jiffies, g=15607817, q=14) Aug 10 19:54:55 ptc_lab_collab3_node17 kernel: Sending NMI from CPU 3 to CPUs 1: Aug 10 19:54:55 ptc_lab_collab3_node17 kernel: Sending NMI from CPU 3 to CPUs 2: Aug 10 19:54:55 ptc_lab_collab3_node17 kernel: NMI backtrace for cpu 2 Aug 10 19:54:55 ptc_lab_collab3_node17 kernel: CPU: 2 PID: 0 Comm: swapper/2 Tainted: G W O 4.19.50-sunxi #5.89 Aug 10 19:54:55 ptc_lab_collab3_node17 kernel: Hardware name: Allwinner sun8i Family Aug 10 19:54:55 ptc_lab_collab3_node17 kernel: PC is at arch_cpu_idle+0x28/0x2c Aug 10 19:54:55 ptc_lab_collab3_node17 kernel: LR is at arch_cpu_idle+0x27/0x2c Aug 10 19:54:55 ptc_lab_collab3_node17 kernel: pc : [<c01078f4>] lr : [<c01078f3>] psr: 40010033 Aug 10 19:54:55 ptc_lab_collab3_node17 kernel: sp : df52dfb0 ip : c0e03d00 fp : 00000000 Aug 10 19:54:55 ptc_lab_collab3_node17 kernel: r10: c0db9870 r9 : c0e04d48 r8 : 00000000 Aug 10 19:54:55 ptc_lab_collab3_node17 kernel: r7 : 00000004 r6 : c0e04db8 r5 : c0e04d70 r4 : ffffe000 Aug 10 19:54:55 ptc_lab_collab3_node17 kernel: r3 : c0116441 r2 : dfb03438 r1 : 0739e3cc r0 : 00000000 Aug 10 19:54:55 ptc_lab_collab3_node17 kernel: Flags: nZcv IRQs on FIQs on Mode SVC_32 ISA Thumb Segment none Aug 10 19:54:55 ptc_lab_collab3_node17 kernel: Control: 50c5387d Table: 5e67806a DAC: 00000051 Aug 10 19:54:55 ptc_lab_collab3_node17 kernel: CPU: 2 PID: 0 Comm: swapper/2 Tainted: G W O 4.19.50-sunxi #5.89 Aug 10 19:54:55 ptc_lab_collab3_node17 kernel: Hardware name: Allwinner sun8i Family Aug 10 19:54:55 ptc_lab_collab3_node17 kernel: [<c010d745>] (unwind_backtrace) from [<c010a2f1>] (show_stack+0x11/0x14) Aug 10 19:54:55 ptc_lab_collab3_node17 kernel: [<c010a2f1>] (show_stack) from [<c08efb01>] (dump_stack+0x69/0x78) Aug 10 19:54:55 ptc_lab_collab3_node17 kernel: [<c08efb01>] (dump_stack) from [<c08f3d4d>] (nmi_cpu_backtrace+0x59/0x90) Aug 10 19:54:55 ptc_lab_collab3_node17 kernel: [<c08f3d4d>] (nmi_cpu_backtrace) from [<c010c5d1>] (handle_IPI+0x85/0x2c0) Aug 10 19:54:55 ptc_lab_collab3_node17 kernel: [<c010c5d1>] (handle_IPI) from [<c05c9287>] (gic_handle_irq+0x67/0x68) Aug 10 19:54:55 ptc_lab_collab3_node17 kernel: [<c05c9287>] (gic_handle_irq) from [<c0101a65>] (__irq_svc+0x65/0x94) Aug 10 19:54:55 ptc_lab_collab3_node17 kernel: Exception stack(0xdf52df60 to 0xdf52dfa8) Aug 10 19:54:55 ptc_lab_collab3_node17 kernel: df60: 00000000 0739e3cc dfb03438 c0116441 ffffe000 c0e04d70 c0e04db8 00000004 Aug 10 19:54:55 ptc_lab_collab3_node17 kernel: df80: 00000000 c0e04d48 c0db9870 00000000 c0e03d00 df52dfb0 c01078f3 c01078f4 Aug 10 19:54:55 ptc_lab_collab3_node17 kernel: dfa0: 40010033 ffffffff Aug 10 19:54:55 ptc_lab_collab3_node17 kernel: [<c0101a65>] (__irq_svc) from [<c01078f4>] (arch_cpu_idle+0x28/0x2c) Aug 10 19:54:55 ptc_lab_collab3_node17 kernel: [<c01078f4>] (arch_cpu_idle) from [<c013e973>] (do_idle+0x14b/0x1d8) Aug 10 19:54:55 ptc_lab_collab3_node17 kernel: [<c013e973>] (do_idle) from [<c013ebf5>] (cpu_startup_entry+0x19/0x1c) Aug 10 19:54:55 ptc_lab_collab3_node17 kernel: [<c013ebf5>] (cpu_startup_entry) from [<401024b1>] (0x401024b1) Aug 10 19:55:01 ptc_lab_collab3_node17 CRON[32672]: pam_unix(cron:account): account root has password changed in future Aug 10 19:55:01 ptc_lab_collab3_node17 CRON[32672]: pam_unix(cron:session): session opened for user root by (uid=0) Aug 10 19:55:01 ptc_lab_collab3_node17 CRON[32673]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1) Aug 10 19:55:01 ptc_lab_collab3_node17 CRON[32672]: pam_unix(cron:session): session closed for user root Aug 10 19:58:32 ptc_lab_collab3_node17 systemd[1]: systemd-udevd.service: Watchdog timeout (limit 3min)! Aug 10 20:00:01 ptc_lab_collab3_node17 CRON[32754]: pam_unix(cron:account): account root has password changed in future Aug 10 20:00:01 ptc_lab_collab3_node17 CRON[32754]: pam_unix(cron:session): session opened for user root by (uid=0) Aug 10 20:00:01 ptc_lab_collab3_node17 CRON[32755]: (root) CMD (/usr/lib/armbian/armbian-truncate-logs) Aug 10 20:00:02 ptc_lab_collab3_node17 systemd[1]: systemd-udevd.service: State 'stop-sigabrt' timed out. Terminating. Aug 10 20:00:06 ptc_lab_collab3_node17 CRON[32754]: pam_unix(cron:session): session closed for user root Aug 10 20:01:33 ptc_lab_collab3_node17 systemd[1]: systemd-udevd.service: State 'stop-sigterm' timed out. Killing. Aug 10 20:03:03 ptc_lab_collab3_node17 systemd[1]: systemd-udevd.service: Processes still around after SIGKILL. Ignoring. Aug 10 20:04:33 ptc_lab_collab3_node17 systemd[1]: systemd-udevd.service: State 'stop-final-sigterm' timed out. Killing. Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[1]: systemd-udevd.service: Processes still around after final SIGKILL. Entering failed mode. Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[1]: systemd-udevd.service: Unit entered failed state. Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[1]: systemd-udevd.service: Failed with result 'timeout'. Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[1]: icecast2.service: Service reached runtime time limit. Stopping. Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[1]: farm-assets.service: Service reached runtime time limit. Stopping. Aug 24 07:45:10 ptc_lab_collab3_node17 CRON[32766]: pam_unix(cron:account): account root has password changed in future Aug 24 07:45:10 ptc_lab_collab3_node17 CRON[32766]: pam_unix(cron:session): session opened for user root by (uid=0) Aug 24 07:45:10 ptc_lab_collab3_node17 CRON[301]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1) Aug 24 07:45:10 ptc_lab_collab3_node17 CRON[32766]: pam_unix(cron:session): session closed for user root Aug 24 07:45:10 ptc_lab_collab3_node17 icecast2[300]: * Stopping streaming media server icecast2 Aug 24 07:45:10 ptc_lab_collab3_node17 CRON[32765]: pam_unix(cron:account): account root has password changed in future Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[1]: Time has been changed Aug 24 07:45:10 ptc_lab_collab3_node17 CRON[32765]: pam_unix(cron:session): session opened for user root by (uid=0) Aug 24 07:45:10 ptc_lab_collab3_node17 icsneo-socketcan[797]: ValueCAN 3 126241 disconnected Aug 24 07:45:10 ptc_lab_collab3_node17 icsneo-socketcan[797]: Removing device mscan_126241 with handle 1 Aug 24 07:45:10 ptc_lab_collab3_node17 kernel: intrepid: Removing device 1 can1 0x17c453e0 Aug 24 07:45:10 ptc_lab_collab3_node17 CRON[309]: (root) CMD (/usr/lib/armbian/armbian-truncate-logs) Aug 24 07:45:10 ptc_lab_collab3_node17 systemctl[308]: Failed to stop assets.mount: Connection timed out Aug 24 07:45:10 ptc_lab_collab3_node17 systemctl[308]: See system logs and 'systemctl status assets.mount' for details. Aug 24 07:45:10 ptc_lab_collab3_node17 systemctl[308]: Failed to get load state of assets.mount: Connection timed out Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[20635]: Time has been changed Aug 24 07:45:10 ptc_lab_collab3_node17 icsneo-socketcan[797]: Removed device mscan_126241 with handle 1, result 0 Aug 24 07:45:10 ptc_lab_collab3_node17 kernel: intrepid: Removed device 1 Aug 24 07:45:10 ptc_lab_collab3_node17 kernel: intrepid: Removing device 0 can0 0x84d5895f Aug 24 07:45:10 ptc_lab_collab3_node17 icsneo-socketcan[797]: Removing device hscan_126241 with handle 0 Aug 24 07:45:10 ptc_lab_collab3_node17 CRON[32765]: pam_unix(cron:session): session closed for user root Aug 24 07:45:10 ptc_lab_collab3_node17 icsneo-socketcan[797]: Removed device hscan_126241 with handle 0, result 0 Aug 24 07:45:10 ptc_lab_collab3_node17 kernel: intrepid: Removed device 0 Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[20635]: Time has been changed Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[20635]: Time has been changed Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[20635]: Time has been changed Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[20635]: Time has been changed Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[20635]: Time has been changed Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[20635]: Time has been changed Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[20635]: Time has been changed Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[20635]: Time has been changed Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[20635]: Time has been changed Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[20635]: Time has been changed Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[20635]: Time has been changed Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[20635]: Time has been changed Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[20635]: Time has been changed Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[20635]: Time has been changed Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[20635]: Time has been changed Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[20635]: Time has been changed Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[20635]: Time has been changed Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[20635]: Time has been changed Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[20635]: Time has been changed Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[20635]: Time has been changed Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[20635]: Time has been changed Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[20635]: Time has been changed Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[20635]: Time has been changed Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[20635]: Time has been changed Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[20635]: Time has been changed Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[20635]: Time has been changed Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[20635]: Time has been changed Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[20635]: Time has been changed Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[20635]: Time has been changed Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[20635]: Time has been changed Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[20635]: Time has been changed Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[20635]: Time has been changed Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[20635]: Time has been changed Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[20635]: Time has been changed Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[20635]: Time has been changed Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[20635]: Time has been changed Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[20635]: Time has been changed Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[20635]: Time has been changed Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[20635]: Time has been changed Aug 24 07:45:10 ptc_lab_collab3_node17 systemd[20635]: Time has been changed Would be nice to receive any suggestions to root cause this issue. Edited August 13, 2019 by Mikhail Kulinich Link to comment Share on other sites More sharing options...
drice Posted August 13, 2019 Share Posted August 13, 2019 7 hours ago, Mikhail Kulinich said: Aug 10 19:54:55 ptc_lab_collab3_node17 kernel: rcu: INFO: rcu_sched detected stalls on CPUs/tasks: Aug 10 19:54:55 ptc_lab_collab3_node17 kernel: rcu: 1-...!: (2 GPs behind) idle=bfa/0/0x1 softirq=4388853/4388853 fqs=0 Aug 10 19:54:55 ptc_lab_collab3_node17 kernel: rcu: 2-...!: (8 GPs behind) idle=114/0/0x0 softirq=4692641/4692641 fqs=0 Aug 10 19:54:55 ptc_lab_collab3_node17 kernel: rcu: (detected by 3, t=39728 jiffies, g=15607817, q=14) Looks like a CPU stall on cores 1 and 2. Nothing obvious in the call stack. The systemd errors are a symptom of the problem, not the root cause of the problem. What type of workload is this system running? See the following link for some possible causes of CPU stalls: https://www.kernel.org/doc/Documentation/RCU/stallwarn.txt Link to comment Share on other sites More sharing options...
Mikhail Kulinich Posted August 13, 2019 Author Share Posted August 13, 2019 4 hours ago, drice said: What type of workload is this system running? The system runs automated tests (using Robot Framework, python tool), interacts via CAN bus (using Intrepid ValueCAN3) with system under test, does some unheavy networking. This was not happening on kernel 4.13.15, but started to happen on 4.19.50. I've separate issue on Github for Intrepid as their driver gives a call stack during boot up. However, not sure if this is related. Link to comment Share on other sites More sharing options...
lomady Posted August 14, 2019 Share Posted August 14, 2019 Link back to similar issue (found by @drice) Link to comment Share on other sites More sharing options...
Sonic Posted March 1, 2020 Share Posted March 1, 2020 Hello! I am experiencing the same issue on OrangePi +2e with: root@orangepiplus2e:/# uname -a Linux orangepiplus2e 4.19.62-sunxi #5.92 SMP Wed Jul 31 22:07:23 CEST 2019 armv7l GNU/Linux It sporadically crashes after about 3 - 5 weeks of running 24h. It is painful because it runs OpenHAB and controls my home. :( It feels like the kernel is OK but it cannot launch any process. Symptoms are: 1. Home automation stops working completely 2. nginx (responsible for externally visible web interface) still works, but returns error 500. 3. The board still responds to PINGs 4. Attempt to connect via ssh fails with "Connection refused" 5. Serial port responds, but only until you press "Enter". The last thing i see is "orangepi TTYs0" line, then silence. No login prompt. 6. SysRq l (show-backtrace-all-active-cpus) outputs nothing. Periodically it dumps: [75677.336667] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [75677.342628] rcu: 2-...!: (10 GPs behind) idle=988/0/0x0 softirq=5109306/5109306 fqs=0 [75677.350661] rcu: 3-...!: (1 GPs behind) idle=b3c/0/0x0 softirq=5733199/5733199 fqs=0 [75677.358594] rcu: (detected by 1, t=5252 jiffies, g=8503225, q=2) [75677.364715] Sending NMI from CPU 1 to CPUs 2: [75677.369403] NMI backtrace for cpu 2 [75677.369411] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.19.62-sunxi #5.92 [75677.369416] Hardware name: Allwinner sun8i Family [75677.369420] PC is at arch_cpu_idle+0x28/0x2c [75677.369424] LR is at arch_cpu_idle+0x27/0x2c [75677.369430] pc : [<c01078f4>] lr : [<c01078f3>] psr: 400f0133 [75677.369434] sp : ee553fb0 ip : 00000000 fp : 00000000 [75677.369439] r10: c0dba870 r9 : c0e04d48 r8 : 00000000 [75677.369445] r7 : 00000004 r6 : c0e04db8 r5 : c0e04d70 r4 : ffffe000 [75677.369450] r3 : c0116441 r2 : eedc2438 r1 : 03d93988 r0 : 00000000 [75677.369456] Flags: nZcv IRQs on FIQs on Mode SVC_32 ISA Thumb Segment none [75677.369461] Control: 50c5387d Table: 6c94c06a DAC: 00000051 [75677.369467] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.19.62-sunxi #5.92 [75677.369471] Hardware name: Allwinner sun8i Family [75677.369478] [<c010d74d>] (unwind_backtrace) from [<c010a2f1>] (show_stack+0x11/0x14) [75677.369484] [<c010a2f1>] (show_stack) from [<c08fc121>] (dump_stack+0x69/0x78) [75677.369490] [<c08fc121>] (dump_stack) from [<c090036d>] (nmi_cpu_backtrace+0x59/0x90) [75677.369497] [<c090036d>] (nmi_cpu_backtrace) from [<c010c5d9>] (handle_IPI+0x85/0x2c0) [75677.369503] [<c010c5d9>] (handle_IPI) from [<c05c9c7f>] (gic_handle_irq+0x67/0x68) [75677.369509] [<c05c9c7f>] (gic_handle_irq) from [<c0101a65>] (__irq_svc+0x65/0x94) [75677.369514] Exception stack(0xee553f60 to 0xee553fa8) [75677.369521] 3f60: 00000000 03d93988 eedc2438 c0116441 ffffe000 c0e04d70 c0e04db8 00000004 [75677.369527] 3f80: 00000000 c0e04d48 c0dba870 00000000 00000000 ee553fb0 c01078f3 c01078f4 [75677.369531] 3fa0: 400f0133 ffffffff [75677.369537] [<c0101a65>] (__irq_svc) from [<c01078f4>] (arch_cpu_idle+0x28/0x2c) [75677.369543] [<c01078f4>] (arch_cpu_idle) from [<c013e96b>] (do_idle+0x14b/0x1d8) [75677.369549] [<c013e96b>] (do_idle) from [<c013ebed>] (cpu_startup_entry+0x19/0x1c) [75677.369555] [<c013ebed>] (cpu_startup_entry) from [<401024b1>] (0x401024b1) [75677.370094] Sending NMI from CPU 1 to CPUs 3: [75677.552324] NMI backtrace for cpu 3 [75677.552331] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.19.62-sunxi #5.92 [75677.552336] Hardware name: Allwinner sun8i Family [75677.552340] PC is at arch_cpu_idle+0x28/0x2c [75677.552345] LR is at arch_cpu_idle+0x27/0x2c [75677.552350] pc : [<c01078f4>] lr : [<c01078f3>] psr: 40070033 [75677.552355] sp : ee555fb0 ip : 7fffffff fp : 00000000 [75677.552360] r10: c0dba870 r9 : c0e04d48 r8 : 00000000 [75677.552365] r7 : 00000008 r6 : c0e04db8 r5 : c0e04d70 r4 : ffffe000 [75677.552371] r3 : c0116441 r2 : eedd4438 r1 : 02b05b3c r0 : 00000000 [75677.552377] Flags: nZcv IRQs on FIQs on Mode SVC_32 ISA Thumb Segment none [75677.552382] Control: 50c5387d Table: 6c4e806a DAC: 00000051 [75677.552388] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.19.62-sunxi #5.92 [75677.552393] Hardware name: Allwinner sun8i Family [75677.552399] [<c010d74d>] (unwind_backtrace) from [<c010a2f1>] (show_stack+0x11/0x14) [75677.552405] [<c010a2f1>] (show_stack) from [<c08fc121>] (dump_stack+0x69/0x78) [75677.552411] [<c08fc121>] (dump_stack) from [<c090036d>] (nmi_cpu_backtrace+0x59/0x90) [75677.552418] [<c090036d>] (nmi_cpu_backtrace) from [<c010c5d9>] (handle_IPI+0x85/0x2c0) [75677.552424] [<c010c5d9>] (handle_IPI) from [<c05c9c7f>] (gic_handle_irq+0x67/0x68) [75677.552430] [<c05c9c7f>] (gic_handle_irq) from [<c0101a65>] (__irq_svc+0x65/0x94) [75677.552435] Exception stack(0xee555f60 to 0xee555fa8) [75677.552442] 5f60: 00000000 02b05b3c eedd4438 c0116441 ffffe000 c0e04d70 c0e04db8 00000008 [75677.552448] 5f80: 00000000 c0e04d48 c0dba870 00000000 7fffffff ee555fb0 c01078f3 c01078f4 [75677.552452] 5fa0: 40070033 ffffffff [75677.552458] [<c0101a65>] (__irq_svc) from [<c01078f4>] (arch_cpu_idle+0x28/0x2c) [75677.552464] [<c01078f4>] (arch_cpu_idle) from [<c013e96b>] (do_idle+0x14b/0x1d8) [75677.552470] [<c013e96b>] (do_idle) from [<c013ebed>] (cpu_startup_entry+0x19/0x1c) [75677.552476] [<c013ebed>] (cpu_startup_entry) from [<401024b1>] (0x401024b1) [75677.553015] rcu: rcu_sched kthread starved for 5305 jiffies! g8503225 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=1 [75677.741423] rcu: RCU grace-period kthread stack dump: [75677.746469] rcu_sched I 0 10 2 0x00000000 [75677.751961] [<c090acdb>] (__schedule) from [<c090b247>] (schedule+0x2f/0x68) [75677.759008] [<c090b247>] (schedule) from [<c090daf7>] (schedule_timeout+0x77/0x320) [75677.766667] [<c090daf7>] (schedule_timeout) from [<c016af6f>] (rcu_gp_kthread+0x41f/0x728) [75677.774933] [<c016af6f>] (rcu_gp_kthread) from [<c0132ae9>] (kthread+0xfd/0x104) [75677.782328] [<c0132ae9>] (kthread) from [<c01010f9>] (ret_from_fork+0x11/0x38) [75677.789545] Exception stack(0xee54bfb0 to 0xee54bff8) [75677.794593] bfa0: 00000000 00000000 00000000 00000000 [75677.802768] bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [75677.810942] bfe0: 00000000 00000000 00000000 00000000 00000013 00000000 I also have SysRq t ("Show task states") output, but it's way too huge to post here. Any recommendations ? Link to comment Share on other sites More sharing options...
Igor Posted March 1, 2020 Share Posted March 1, 2020 1 hour ago, Sonic said: It sporadically crashes after about 3 - 5 weeks of running 24h. It is painful because it runs OpenHAB and controls my home. 1 hour ago, Sonic said: Any recommendations ? Link to comment Share on other sites More sharing options...
Agay1 Posted March 12, 2020 Share Posted March 12, 2020 Mikhail have you found a solution? I'm really interested as i have the same issue. Igor, thanks a lot for the helpful link with side dish. may i ask you questions in case i'm going to have some? i'm just trying to find out what's wrong... Link to comment Share on other sites More sharing options...
Recommended Posts