bozzio Posted July 13, 2020 Posted July 13, 2020 (edited) Armbianmonitor: http://ix.io/2ryU I'am using Cubieboard2. Todays morning i've got mistery "rcu: INFO: rcu_sched detected stalls on CPUs/tasks" with a complete failure of the entire network subsystem. How it was: Jul 13 09:12:36 hassio kernel: rcu: INFO: rcu_sched detected stalls on CPUs/tasks: Jul 13 09:12:37 hassio kernel: rcu: 1-...0: (21 ticks this GP) idle=a56/1/0x40000000 softirq=4946049/4946050 fqs=2286 Jul 13 09:12:37 hassio kernel: (detected by 0, t=5252 jiffies, g=11125849, q=117) Jul 13 09:12:37 hassio kernel: Sending NMI from CPU 0 to CPUs 1: Jul 13 09:13:13 hassio systemd[1]: systemd-udevd.service: Watchdog timeout (limit 3min)! Jul 13 09:13:13 hassio systemd[1]: systemd-udevd.service: Killing process 308 (systemd-udevd) with signal SIGABRT. Jul 13 09:13:39 hassio kernel: rcu: INFO: rcu_sched detected stalls on CPUs/tasks: Jul 13 09:13:39 hassio kernel: rcu: 1-...0: (21 ticks this GP) idle=a56/1/0x40000000 softirq=4946049/4946050 fqs=8014 Jul 13 09:13:39 hassio kernel: (detected by 0, t=21007 jiffies, g=11125849, q=2391) Jul 13 09:13:40 hassio kernel: Sending NMI from CPU 0 to CPUs 1: Jul 13 09:13:40 hassio kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 1-... } 6684 jiffies s: 13809 root: 0x2/. Jul 13 09:13:40 hassio kernel: rcu: blocking rcu_node structures: Jul 13 09:13:40 hassio kernel: Task dump for CPU 1: Jul 13 09:13:40 hassio kernel: kworker/1:0 R running task 0 25335 2 0x00000002 Jul 13 09:13:40 hassio kernel: Workqueue: events dbs_work_handler Jul 13 09:13:40 hassio kernel: [<c086fd9f>] (__schedule) from [<00000008>] (0x8) Jul 13 09:14:43 hassio kernel: rcu: INFO: rcu_sched detected stalls on CPUs/tasks: Jul 13 09:14:43 hassio kernel: rcu: 1-...0: (21 ticks this GP) idle=a56/1/0x40000000 softirq=4946049/4946050 fqs=13734 Jul 13 09:14:43 hassio kernel: (detected by 0, t=36762 jiffies, g=11125849, q=5496) Jul 13 09:14:43 hassio kernel: Sending NMI from CPU 0 to CPUs 1: Jul 13 09:14:42 hassio systemd[1]: snapd.service: Watchdog timeout (limit 5min)! Jul 13 09:14:42 hassio systemd[1]: snapd.service: Killing process 637 (snapd) with signal SIGABRT. Jul 13 09:14:43 hassio systemd[1]: systemd-udevd.service: State 'stop-watchdog' timed out. Terminating. ... Jul 13 09:15:47 hassio kernel: [<c086fd9f>] (__schedule) from [<00000008>] (0x8) Jul 13 09:15:48 hassio udevd[189]: worker [1659] /devices/virtual/block/loop1 is taking a long time Jul 13 09:15:48 hassio udevd[493]: worker [567] /devices/virtual/block/loop1 is taking a long time Jul 13 09:15:49 hassio udevd[182]: worker [31741] /devices/virtual/block/loop1 is taking a long time Jul 13 09:16:06 hassio kernel: ------------[ cut here ]------------ Jul 13 09:16:06 hassio kernel: WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:448 dev_watchdog+0x213/0x214 Jul 13 09:16:06 hassio kernel: NETDEV WATCHDOG: eth0 (sun7i-dwmac): transmit queue 0 timed out Jul 13 09:16:06 hassio kernel: Modules linked in: tcp_diag inet_diag xt_nat xt_tcpudp veth xt_conntrack nf_conntrack_netlink nfnetlink xt_addrtype iptable_filter br_netfilter bridge stp llc aufs xt_MASQUERADE iptable_nat > Jul 13 09:16:06 hassio kernel: CPU: 0 PID: 0 Comm: swapper/0 Tainted: G C 5.4.45-sunxi #20.05.4 Jul 13 09:16:06 hassio kernel: Hardware name: Allwinner sun7i (A20) Family Jul 13 09:16:06 hassio kernel: [<c010dc49>] (unwind_backtrace) from [<c010a245>] (show_stack+0x11/0x14) Jul 13 09:16:06 hassio kernel: [<c010a245>] (show_stack) from [<c085f97b>] (dump_stack+0x6f/0x7c) Jul 13 09:16:06 hassio kernel: [<c085f97b>] (dump_stack) from [<c011e3cb>] (__warn+0xb7/0xb8) Jul 13 09:16:06 hassio kernel: [<c011e3cb>] (__warn) from [<c011e65b>] (warn_slowpath_fmt+0x53/0x5c) Jul 13 09:16:06 hassio kernel: [<c011e65b>] (warn_slowpath_fmt) from [<c076eae3>] (dev_watchdog+0x213/0x214) Jul 13 09:16:06 hassio kernel: [<c076eae3>] (dev_watchdog) from [<c017299b>] (call_timer_fn+0x27/0x128) Jul 13 09:16:06 hassio kernel: [<c017299b>] (call_timer_fn) from [<c01733e5>] (run_timer_softirq+0x3f5/0x414) Jul 13 09:16:06 hassio kernel: [<c01733e5>] (run_timer_softirq) from [<c01022f7>] (__do_softirq+0xdf/0x288) Jul 13 09:16:06 hassio kernel: [<c01022f7>] (__do_softirq) from [<c0122f4b>] (irq_exit+0x7b/0x90) Jul 13 09:16:06 hassio kernel: [<c0122f4b>] (irq_exit) from [<c0161523>] (__handle_domain_irq+0x47/0x84) Jul 13 09:16:06 hassio kernel: [<c0161523>] (__handle_domain_irq) from [<c0517211>] (gic_handle_irq+0x39/0x6c) Jul 13 09:16:06 hassio kernel: [<c0517211>] (gic_handle_irq) from [<c0101ae5>] (__irq_svc+0x65/0x94) Jul 13 09:16:06 hassio kernel: Exception stack(0xc0e01f30 to 0xc0e01f78) Jul 13 09:16:06 hassio kernel: 1f20: 00000000 129145fc ef691034 c01164c1 Jul 13 09:16:06 hassio kernel: 1f40: ffffe000 c0e04fa4 c0e04fec 00000001 00000000 c0db99f0 c0ec4b25 00000000 Jul 13 09:16:06 hassio kernel: 1f60: c0f0cf98 c0e01f80 c0107c6f c0107c70 40000033 ffffffff Jul 13 09:16:06 hassio kernel: [<c0101ae5>] (__irq_svc) from [<c0107c70>] (arch_cpu_idle+0x28/0x2c) Jul 13 09:16:06 hassio kernel: [<c0107c70>] (arch_cpu_idle) from [<c014126b>] (do_idle+0x143/0x1b0) Jul 13 09:16:06 hassio kernel: [<c014126b>] (do_idle) from [<c01414b9>] (cpu_startup_entry+0x19/0x20) Jul 13 09:16:06 hassio kernel: [<c01414b9>] (cpu_startup_entry) from [<c0d00c81>] (start_kernel+0x3eb/0x3f8) Jul 13 09:16:06 hassio kernel: ---[ end trace 4f6c33f5bb7f39ba ]--- Jul 13 09:16:06 hassio kernel: sun7i-dwmac 1c50000.ethernet eth0: Reset adapter. Jul 13 09:16:13 hassio systemd[1]: systemd-udevd.service: State 'stop-sigterm' timed out. Killing. Jul 13 09:16:13 hassio systemd[1]: systemd-udevd.service: Killing process 308 (systemd-udevd) with signal SIGKILL. Jul 13 09:16:14 hassio systemd[1]: snapd.service: start operation timed out. Terminating. I've read https://www.kernel.org/doc/Documentation/RCU/stallwarn.txt and many similar posts on this forum. I will try to cut cpu frequency with cpufreq-set . My question is: How to detect magic string "rcu: INFO: rcu_sched detected stalls on CPUs/tasks" and reboot system instantly? Edited July 13, 2020 by bozzio
Recommended Posts