OrangePi+ 2E (H3) crashes on 4.14 with INFO: rcu_sched self-detected stall on CPU

[René Kliment]

Hello everyone.
This happens on my OrangePi after a couple of hours of running. I don't think this happened before installing the kernel 4.14. I haven't changed my power supply, which should be working well - custom DC/DC converter connected to GPIO pins.

Is anyone else experiencing this? Anything I can do?
 

[19155.760977] INFO: rcu_sched self-detected stall on CPU
[19155.760983] INFO: rcu_sched self-detected stall on CPU
[19155.760997] INFO: rcu_sched detected stalls on CPUs/tasks:
[19155.761010]  2-...: (1 ticks this GP) idle=18a/1/0 softirq=332295/332295 fqs=0
[19155.761017]  0-...: (0 ticks this GP) idle=dbe/1/0 softirq=475103/475103 fqs=0
[19155.761018]
[19155.761025]  1-...: (1 GPs behind) idle=f36/1/0 softirq=422883/422884 fqs=0
[19155.761034]  2-...: (1 ticks this GP) idle=18a/1/0 softirq=332295/332295 fqs=0
[19155.761040] rcu_sched kthread starved for 91911 jiffies! g126395 c126394 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x402 ->cpu=3
[19155.761041]
[19155.819568]  1-...: (1 GPs behind) idle=f36/1/0 softirq=422883/422884 fqs=0
[19155.826612]   (t=91911 jiffies g=126395 c=126394 q=3)
[19155.831672] rcu_sched kthread starved for 91911 jiffies! g126395 c126394 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x0 ->cpu=0
[19185.762182] rcu_sched kthread starved for 91911 jiffies! g126395 c126394 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x0 ->cpu=0
[19155.760977] INFO: rcu_sched self-detected stall on CPU
[19155.760983] INFO: rcu_sched self-detected stall on CPU
[19155.760997] INFO: rcu_sched detected stalls on CPUs/tasks:
[19155.761010]  2-...: (1 ticks this GP) idle=18a/1/0 softirq=332295/332295 fqs=0 
[19155.761017]  0-...: (0 ticks this GP) idle=dbe/1/0 softirq=475103/475103 fqs=0 
[19155.761018] 
[19155.761025]  1-...: (1 GPs behind) idle=f36/1/0 softirq=422883/422884 fqs=0 
[19155.761028]  (t=91911 jiffies g=126395 c=126394 q=3)
[19155.761034]  2-...: (1 ticks this GP) idle=18a/1/0 softirq=332295/332295 fqs=0 
[19155.761040] rcu_sched kthread starved for 91911 jiffies! g126395 c126394 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x402 ->cpu=3
[19155.761041] 
[19155.761044] rcu_sched       I
[19155.761048] (detected by 3, t=91911 jiffies, g=126395, c=126394, q=3)
[19155.761052]     0     8      2 0x00000000
[19155.761057] Sending NMI from CPU 3 to CPUs 0:
[19155.761079] [<c08762b9>] (__schedule) from [<c087682b>] (schedule+0x2f/0x68)
[19155.761093] [<c087682b>] (schedule) from [<c087909d>] (schedule_timeout+0x75/0x314)
[19155.761108] [<c087909d>] (schedule_timeout) from [<c0164f95>] (rcu_gp_kthread+0x415/0x674)
[19155.761123] [<c0164f95>] (rcu_gp_kthread) from [<c0131a9d>] (kthread+0xfd/0x104)
[19155.761138] [<c0131a9d>] (kthread) from [<c0106719>] (ret_from_fork+0x11/0x38)
[19155.819568]  1-...: (1 GPs behind) idle=f36/1/0 softirq=422883/422884 fqs=0 
[19155.826612]   (t=91911 jiffies g=126395 c=126394 q=3)
[19155.831672] rcu_sched kthread starved for 91911 jiffies! g126395 c126394 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x0 ->cpu=0
[19155.842191] rcu_sched       R  running task        0     8      2 0x00000000
[19155.842210] [<c08762b9>] (__schedule) from [<c087682b>] (schedule+0x2f/0x68)
[19155.842222] [<c087682b>] (schedule) from [<c087909d>] (schedule_timeout+0x75/0x314)
[19155.842234] [<c087909d>] (schedule_timeout) from [<c0164f95>] (rcu_gp_kthread+0x415/0x674)
[19155.842245] [<c0164f95>] (rcu_gp_kthread) from [<c0131a9d>] (kthread+0xfd/0x104)
[19155.842256] [<c0131a9d>] (kthread) from [<c0106719>] (ret_from_fork+0x11/0x38)
[19165.761432] Sending NMI from CPU 3 to CPUs 1:
[19175.761806] Sending NMI from CPU 3 to CPUs 2:
[19185.762182] rcu_sched kthread starved for 91911 jiffies! g126395 c126394 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x0 ->cpu=0
[19185.762186] Sending NMI from CPU 2 to CPUs 0:
[19185.772707] rcu_sched       R  running task        0     8      2 0x00000000
[19185.772726] [<c08762b9>] (__schedule) from [<c087682b>] (schedule+0x2f/0x68)
[19185.772738] [<c087682b>] (schedule) from [<c087909d>] (schedule_timeout+0x75/0x314)
[19185.772749] [<c087909d>] (schedule_timeout) from [<c0164f95>] (rcu_gp_kthread+0x415/0x674)
[19185.772760] [<c0164f95>] (rcu_gp_kthread) from [<c0131a9d>] (kthread+0xfd/0x104)
[19185.772772] [<c0131a9d>] (kthread) from [<c0106719>] (ret_from_fork+0x11/0x38)
[19195.762559] Sending NMI from CPU 2 to CPUs 1:
[19205.762933] NMI backtrace for cpu 2
[19205.762942] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.14.14-sunxi #38
[19205.762945] Hardware name: Allwinner sun8i Family
[19205.762959] [<c010db15>] (unwind_backtrace) from [<c010a0d9>] (show_stack+0x11/0x14)
[19205.762970] [<c010a0d9>] (show_stack) from [<c0867e29>] (dump_stack+0x69/0x78)
[19205.762983] [<c0867e29>] (dump_stack) from [<c086bc37>] (nmi_cpu_backtrace+0xd3/0xd4)
[19205.762996] [<c086bc37>] (nmi_cpu_backtrace) from [<c086bccf>] (nmi_trigger_cpumask_backtrace+0x97/0xd0)
[19205.763008] [<c086bccf>] (nmi_trigger_cpumask_backtrace) from [<c01662a7>] (rcu_dump_cpu_stacks+0x77/0x94)
[19205.763019] [<c01662a7>] (rcu_dump_cpu_stacks) from [<c0165a65>] (rcu_check_callbacks+0x4d5/0x690)
[19205.763032] [<c0165a65>] (rcu_check_callbacks) from [<c0169f5f>] (update_process_times+0x2b/0x48)
[19205.763046] [<c0169f5f>] (update_process_times) from [<c0177d11>] (tick_sched_timer+0x31/0x68)
[19205.763057] [<c0177d11>] (tick_sched_timer) from [<c016ac85>] (__hrtimer_run_queues+0xf5/0x224)
[19205.763066] [<c016ac85>] (__hrtimer_run_queues) from [<c016af81>] (hrtimer_interrupt+0x81/0x180)
[19205.763079] [<c016af81>] (hrtimer_interrupt) from [<c07483a1>] (arch_timer_handler_phys+0x25/0x28)
[19205.763093] [<c07483a1>] (arch_timer_handler_phys) from [<c015d52f>] (handle_percpu_devid_irq+0x57/0x19c)
[19205.763107] [<c015d52f>] (handle_percpu_devid_irq) from [<c0159a49>] (generic_handle_irq+0x1d/0x28)
[19205.763120] [<c0159a49>] (generic_handle_irq) from [<c0159e59>] (__handle_domain_irq+0x45/0x84)
[19205.763131] [<c0159e59>] (__handle_domain_irq) from [<c01013b5>] (gic_handle_irq+0x39/0x68)
[19205.763141] [<c01013b5>] (gic_handle_irq) from [<c010aa25>] (__irq_svc+0x65/0x94)
[19205.763146] Exception stack(0xee523f78 to 0xee523fc0)
[19205.763152] 3f60:                                                       00000001 00000000
[19205.763161] 3f80: 00000000 c0116561 ffffe000 c0d03fcc c0d03f6c c0cb6438 c0ddd8eb 00000000
[19205.763170] 3fa0: 00000000 00000000 00087b5d ee523fc8 c01070e7 c01070e8 40000033 ffffffff
[19205.763181] [<c010aa25>] (__irq_svc) from [<c01070e8>] (arch_cpu_idle+0x28/0x2c)
[19205.763194] [<c01070e8>] (arch_cpu_idle) from [<c014c93d>] (do_idle+0x115/0x16c)
[19205.763206] [<c014c93d>] (do_idle) from [<c014cb89>] (cpu_startup_entry+0x19/0x1c)
[19205.763216] [<c014cb89>] (cpu_startup_entry) from [<401016f1>] (0x401016f1)
[19205.763226] Sending NMI from CPU 1 to CPUs 0:
[19215.763599] NMI backtrace for cpu 1
[19215.763605] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.14.14-sunxi #38
[19215.763608] Hardware name: Allwinner sun8i Family
[19215.763618] [<c010db15>] (unwind_backtrace) from [<c010a0d9>] (show_stack+0x11/0x14)
[19215.763627] [<c010a0d9>] (show_stack) from [<c0867e29>] (dump_stack+0x69/0x78)
[19215.763638] [<c0867e29>] (dump_stack) from [<c086bc37>] (nmi_cpu_backtrace+0xd3/0xd4)
[19215.763650] [<c086bc37>] (nmi_cpu_backtrace) from [<c086bccf>] (nmi_trigger_cpumask_backtrace+0x97/0xd0)
[19215.763661] [<c086bccf>] (nmi_trigger_cpumask_backtrace) from [<c01662a7>] (rcu_dump_cpu_stacks+0x77/0x94)
[19215.763672] [<c01662a7>] (rcu_dump_cpu_stacks) from [<c0165a65>] (rcu_check_callbacks+0x4d5/0x690)
[19215.763683] [<c0165a65>] (rcu_check_callbacks) from [<c0169f5f>] (update_process_times+0x2b/0x48)
[19215.763695] [<c0169f5f>] (update_process_times) from [<c0177d11>] (tick_sched_timer+0x31/0x68)
[19215.763705] [<c0177d11>] (tick_sched_timer) from [<c016ac85>] (__hrtimer_run_queues+0xf5/0x224)
[19215.763714] [<c016ac85>] (__hrtimer_run_queues) from [<c016af81>] (hrtimer_interrupt+0x81/0x180)
[19215.763724] [<c016af81>] (hrtimer_interrupt) from [<c07483a1>] (arch_timer_handler_phys+0x25/0x28)
[19215.763735] [<c07483a1>] (arch_timer_handler_phys) from [<c015d52f>] (handle_percpu_devid_irq+0x57/0x19c)
[19215.763746] [<c015d52f>] (handle_percpu_devid_irq) from [<c0159a49>] (generic_handle_irq+0x1d/0x28)
[19215.763758] [<c0159a49>] (generic_handle_irq) from [<c0159e59>] (__handle_domain_irq+0x45/0x84)
[19215.763768] [<c0159e59>] (__handle_domain_irq) from [<c01013b5>] (gic_handle_irq+0x39/0x68)
[19215.763777] [<c01013b5>] (gic_handle_irq) from [<c010aa25>] (__irq_svc+0x65/0x94)
[19215.763781] Exception stack(0xee521f78 to 0xee521fc0)
[19215.763787] 1f60:                                                       00000001 00000000
[19215.763796] 1f80: 00000000 c0116561 ffffe000 c0d03fcc c0d03f6c c0cb6438 c0ddd8eb 00000000
[19215.763805] 1fa0: 00000000 00000000 00000018 ee521fc8 c01070e7 c01070e8 40000033 ffffffff
[19215.763815] [<c010aa25>] (__irq_svc) from [<c01070e8>] (arch_cpu_idle+0x28/0x2c)
[19215.763826] [<c01070e8>] (arch_cpu_idle) from [<c014c93d>] (do_idle+0x115/0x16c)
[19215.763837] [<c014c93d>] (do_idle) from [<c014cb89>] (cpu_startup_entry+0x19/0x1c)
[19215.763846] [<c014cb89>] (cpu_startup_entry) from [<401016f1>] (0x401016f1)
[19215.763853] Sending NMI from CPU 1 to CPUs 2:
[19225.764386] NMI backtrace for cpu 2
[19225.764393] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.14.14-sunxi #38
[19225.764395] Hardware name: Allwinner sun8i Family
[19225.764399] task: ee4f8000 task.stack: ee522000
[19225.764404] PC is at __do_softirq+0x7a/0x25c
[19225.764412] LR is at irq_exit+0x7f/0xc4
[19225.764416] pc : [<c0101462>]    lr : [<c011ef33>]    psr: 40000133
[19225.764419] sp : ee523ee8  ip : 7fffffff  fp : 4000001f
[19225.764422] r10: c0d02080  r9 : ee434000  r8 : 00000001
[19225.764426] r7 : ffffe000  r6 : 00000282  r5 : 00000000  r4 : ffffe000
[19225.764430] r3 : 00000080  r2 : 00000000  r1 : c0df6740  r0 : c0df6740
[19225.764435] Flags: nZcv  IRQs on  FIQs on  Mode SVC_32  ISA Thumb  Segment none
[19225.764439] Control: 50c5387d  Table: 56ff006a  DAC: 00000051
[19225.764444] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.14.14-sunxi #38
[19225.764446] Hardware name: Allwinner sun8i Family
[19225.764456] [<c010db15>] (unwind_backtrace) from [<c010a0d9>] (show_stack+0x11/0x14)
[19225.764465] [<c010a0d9>] (show_stack) from [<c0867e29>] (dump_stack+0x69/0x78)
[19225.764475] [<c0867e29>] (dump_stack) from [<c086bbef>] (nmi_cpu_backtrace+0x8b/0xd4)
[19225.764486] [<c086bbef>] (nmi_cpu_backtrace) from [<c010ca01>] (handle_IPI+0x75/0x278)
[19225.764495] [<c010ca01>] (handle_IPI) from [<c01013e3>] (gic_handle_irq+0x67/0x68)
[19225.764502] [<c01013e3>] (gic_handle_irq) from [<c010aa25>] (__irq_svc+0x65/0x94)
[19225.764506] Exception stack(0xee523e98 to 0xee523ee0)
[19225.764510] 3e80:                                                       c0df6740 c0df6740
[19225.764519] 3ea0: 00000000 00000080 ffffe000 00000000 00000282 ffffe000 00000001 ee434000
[19225.764527] 3ec0: c0d02080 4000001f 7fffffff ee523ee8 c011ef33 c0101462 40000133 ffffffff
[19225.764535] [<c010aa25>] (__irq_svc) from [<c0101462>] (__do_softirq+0x7a/0x25c)
[19225.764545] [<c0101462>] (__do_softirq) from [<c011ef33>] (irq_exit+0x7f/0xc4)
[19225.764557] [<c011ef33>] (irq_exit) from [<c0159e5d>] (__handle_domain_irq+0x49/0x84)
[19225.764567] [<c0159e5d>] (__handle_domain_irq) from [<c01013b5>] (gic_handle_irq+0x39/0x68)
[19225.764575] [<c01013b5>] (gic_handle_irq) from [<c010aa25>] (__irq_svc+0x65/0x94)
[19225.764578] Exception stack(0xee523f78 to 0xee523fc0)
[19225.764582] 3f60:                                                       00000001 00000000
[19225.764590] 3f80: 00000000 c0116561 ffffe000 c0d03fcc c0d03f6c c0cb6438 c0ddd8eb 00000000
[19225.764603] 3fa0: 00000000 00000000 00087b5d ee523fc8 c01070e7 c01070e8 40000033 ffffffff
[19225.764613] [<c010aa25>] (__irq_svc) from [<c01070e8>] (arch_cpu_idle+0x28/0x2c)
[19225.764624] [<c01070e8>] (arch_cpu_idle) from [<c014c93d>] (do_idle+0x115/0x16c)
[19225.764642] [<c014c93d>] (do_idle) from [<c014cb89>] (cpu_startup_entry+0x19/0x1c)
[19225.764656] [<c014cb89>] (cpu_startup_entry) from [<401016f1>] (0x401016f1)

 

[Igor]

18 hours ago, René Kliment said:

Is anyone else experiencing this? Anything I can do?

No, unknown to me. What kind of things were you doing on it? Or idle?

[René Kliment]

3 hours ago, Igor said:

No, unknown to me. What kind of things were you doing on it? Or idle?

Happens with both idle and busy Pi. I've been trying to rsync+ssh from a remote to an USB connected HDD. The HDD has its own power.
Also, the Pi has a heat sink, so no overheating either.

This is the weirdest thing ...

[sermore]

Hi everybody,

 

Same behavior here for a Lime2 running latest kernel:

Welcome to ARMBIAN 5.38 stable Debian GNU/Linux 8 (jessie) 4.14.15-sunxi

After around 4 hours of idle, this errors start appearing crashing the board after a while.

 

[    0.430765] cpuidle: using governor ladder
[    0.430826] cpuidle: using governor menu
[    0.432049] hw-breakpoint: found 5 (+1 reserved) breakpoint and 4 watchpoint registers.
[    0.432085] hw-breakpoint: maximum watchpoint size is 8 bytes.
[    0.616165] raid6: int32x1  gen()   171 MB/s
[    0.786408] raid6: int32x1  xor()   145 MB/s
[    0.956533] raid6: int32x2  gen()   231 MB/s
[    1.126539] raid6: int32x2  xor()   175 MB/s
[    1.296737] raid6: int32x4  gen()   233 MB/s
[    1.466834] raid6: int32x4  xor()   174 MB/s
[    1.637134] raid6: int32x8  gen()   223 MB/s
[    1.807109] raid6: int32x8  xor()   155 MB/s
[    1.977297] raid6: neonx1   gen()   447 MB/s
[    2.147379] raid6: neonx1   xor()   417 MB/s
[    2.317515] raid6: neonx2   gen()   600 MB/s
[    2.487619] raid6: neonx2   xor()   537 MB/s
[    2.657756] raid6: neonx4   gen()   721 MB/s
[    2.827934] raid6: neonx4   xor()   606 MB/s
[    2.998142] raid6: neonx8   gen()   650 MB/s
[    3.168205] raid6: neonx8   xor()   548 MB/s
[    3.168222] raid6: using algorithm neonx4 gen() 721 MB/s
[    3.168236] raid6: .... xor() 606 MB/s, rmw enabled
[    3.168251] raid6: using neon recovery algorithm
[    3.168680] reg-fixed-voltage ahci-5v: could not find pctldev for node /soc@01c00000/pinctrl@01c20800/ahci_pwr_pin@1, deferring probe
[    3.168761] reg-fixed-voltage usb0-vbus: could not find pctldev for node /soc@01c00000/pinctrl@01c20800/usb0_vbus_pin@0, deferring probe
[    3.170286] SCSI subsystem initialized
[    3.170635] libata version 3.00 loaded.
[    3.170877] usbcore: registered new interface driver usbfs
[    3.170947] usbcore: registered new interface driver hub
[    3.171023] usbcore: registered new device driver usb
[    3.171233] media: Linux media interface: v0.10
[    3.171278] Linux video capture interface: v2.00
[    3.171354] pps_core: LinuxPPS API ver. 1 registered
[    3.171370] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    3.171406] PTP clock support registered
[    3.173002] clocksource: Switched to clocksource arch_sys_counter
[    3.276824] VFS: Disk quotas dquot_6.6.0
[    3.276954] VFS: Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
[    3.285576] NET: Registered protocol family 2
[    3.286305] TCP established hash table entries: 8192 (order: 3, 32768 bytes)
[    3.286426] TCP bind hash table entries: 8192 (order: 4, 65536 bytes)
[    3.286563] TCP: Hash tables configured (established 8192 bind 8192)
[    3.286699] UDP hash table entries: 512 (order: 2, 16384 bytes)
[    3.286774] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
[    3.287025] NET: Registered protocol family 1
[    3.287604] RPC: Registered named UNIX socket transport module.
[    3.287636] RPC: Registered udp transport module.
[    3.287650] RPC: Registered tcp transport module.
[    3.287664] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    3.287966] Trying to unpack rootfs image as initramfs...
[    3.602626] Freeing initrd memory: 4468K
[    3.603579] hw perfevents: no interrupt-affinity property for /pmu, guessing.
[    3.603963] hw perfevents: enabled with armv7_cortex_a7 PMU driver, 5 counters available
[    3.605010] audit: initializing netlink subsys (disabled)
[    3.605314] audit: type=2000 audit(3.590:1): state=initialized audit_enabled=0 res=1
[    3.605707] Initialise system trusted keyrings
[    3.605946] workingset: timestamp_bits=14 max_order=18 bucket_order=4
[    3.611182] zbud: loaded
[    3.613784] NFS: Registering the id_resolver key type
[    3.613844] Key type id_resolver registered
[    3.613860] Key type id_legacy registered
[    3.613886] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    3.613905] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[    3.615061] JFS: nTxBlock = 7996, nTxLock = 63973
[    3.625790] SGI XFS with ACLs, security attributes, realtime, no debug enabled
[    3.633462] Key type asymmetric registered
[    3.633577] bounce: pool size: 64 pages
[    3.633686] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247)
[    3.633890] io scheduler noop registered
[    3.633912] io scheduler deadline registered
[    3.634244] io scheduler cfq registered (default)
[    3.634264] io scheduler mq-deadline registered
[    3.634279] io scheduler kyber registered
[    3.634427] io scheduler bfq registered
[    3.635405] sun4i-usb-phy 1c13400.phy: could not find pctldev for node /soc@01c00000/pinctrl@01c20800/usb0_id_detect_pin@0, deferring probe
[    3.638749] sun4i-pinctrl 1c20800.pinctrl: initialized sunXi PIO driver
[    3.694001] Serial: 8250/16550 driver, 8 ports, IRQ sharing disabled
[    3.697445] console [ttyS0] disabled
[    3.717691] 1c28000.serial: ttyS0 at MMIO 0x1c28000 (irq = 43, base_baud = 1500000) is a U6_16550A
[    4.622634] console [ttyS0] enabled
[    4.630296] brd: module loaded
[    4.639839] loop: module loaded
[    4.645140] libphy: Fixed MDIO Bus: probed
[    4.649851] sun7i-dwmac 1c50000.ethernet: PTP uses main clock
[    4.655654] sun7i-dwmac 1c50000.ethernet: no reset control found
[    4.661705] sun7i-dwmac 1c50000.ethernet: no regulator found
[    4.667437] sun7i-dwmac 1c50000.ethernet: Ring mode enabled
[    4.673032] sun7i-dwmac 1c50000.ethernet: DMA HW capability register supported
[    4.680260] sun7i-dwmac 1c50000.ethernet: Normal descriptors
[    4.699525] libphy: stmmac: probed
[    4.702968] RTL8211B Gigabit Ethernet stmmac-0:01: attached PHY driver [RTL8211B Gigabit Ethernet] (mii_bus:phy_addr=stmmac-0:01, irq=POLL)
[    4.716645] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    4.723219] ehci-platform: EHCI generic platform driver
[    4.728865] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    4.735140] ohci-platform: OHCI generic platform driver
[    4.740799] usbcore: registered new interface driver usb-storage
[    4.747567] mousedev: PS/2 mouse device common for all mice
[    4.753996] sunxi-rtc 1c20d00.rtc: rtc core: registered rtc-sunxi as rtc0
[    4.760803] sunxi-rtc 1c20d00.rtc: RTC enabled
[    4.765428] i2c /dev entries driver
[    4.769889] axp20x-i2c 0-0034: AXP20x variant AXP209 found
[    4.783238] axp20x-gpio axp20x-gpio: AXP209 GPIO driver loaded
[    4.797380] input: axp20x-pek as /devices/platform/soc@01c00000/1c2ac00.i2c/i2c-0/0-0034/axp20x-pek/input/input0
[    4.808304] ldo1: supplied by regulator-dummy
[    4.812951] ldo2: supplied by regulator-dummy
[    4.818454] ldo3: supplied by regulator-dummy
[    4.823495] ldo4: supplied by regulator-dummy
[    4.828061] ldo5: supplied by regulator-dummy
[    4.833067] dcdc2: supplied by regulator-dummy
[    4.838195] dcdc3: supplied by regulator-dummy
[    4.846580] axp20x-i2c 0-0034: Backup (RTC) battery charging is disabled
[    4.853467] axp20x-i2c 0-0034: AXP20X driver loaded
[    4.861403] sunxi-wdt 1c20c90.watchdog: Watchdog enabled (timeout=16 sec, nowayout=0)
[    4.881744] sunxi-mmc 1c0f000.mmc: Got CD GPIO
[    4.944802] sunxi-mmc 1c0f000.mmc: base:0xf0f6a000 irq:27
[    4.963850] ledtrig-cpu: registered to indicate activity on CPUs
[    4.969993] hidraw: raw HID events driver (C) Jiri Kosina
[    4.975576] usbcore: registered new interface driver usbhid
[    4.981167] usbhid: USB HID core driver
[    4.993241] NET: Registered protocol family 10
[    5.028107] Segment Routing with IPv6
[    5.031942] NET: Registered protocol family 17
[    5.036712] Key type dns_resolver registered
[    5.041405] Registering SWP/SWPB emulation handler
[    5.047011] registered taskstats version 1
[    5.051146] Loading compiled-in X.509 certificates
[    5.056181] zswap: loaded using pool lzo/zbud
[    5.062555] Btrfs loaded, crc32c=crc32c-generic
[    5.071664] mmc0: host does not support reading read-only switch, assuming write-enable
[    5.076538] Key type encrypted registered
[    5.086486] mmc0: new high speed SDHC card at address 0002
[    5.092664] mmcblk0: mmc0:0002 00000 3.70 GiB 
[    5.098869]  mmcblk0: p1
[    5.183244] ahci-sunxi 1c18000.sata: controller can't do PMP, turning off CAP_PMP
[    5.190874] ahci-sunxi 1c18000.sata: SSS flag set, parallel bus scan disabled
[    5.198182] ahci-sunxi 1c18000.sata: AHCI 0001.0100 32 slots 1 ports 3 Gbps 0x1 impl platform mode
[    5.207249] ahci-sunxi 1c18000.sata: flags: ncq sntf stag pm led clo only pio slum part ccc 
[    5.233859] scsi host0: ahci-sunxi
[    5.237530] ata1: SATA max UDMA/133 mmio [mem 0x01c18000-0x01c18fff] port 0x100 irq 33
[    5.246127] ehci-platform 1c14000.usb: EHCI Host Controller
[    5.251749] ehci-platform 1c14000.usb: new USB bus registered, assigned bus number 1
[    5.259836] ehci-platform 1c14000.usb: irq 30, io mem 0x01c14000
[    5.293036] ehci-platform 1c14000.usb: USB 2.0 started, EHCI 1.00
[    5.299406] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[    5.306229] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    5.313467] usb usb1: Product: EHCI Host Controller
[    5.318348] usb usb1: Manufacturer: Linux 4.14.15-sunxi ehci_hcd
[    5.324367] usb usb1: SerialNumber: 1c14000.usb
[    5.334934] hub 1-0:1.0: USB hub found
[    5.338765] hub 1-0:1.0: 1 port detected
[    5.353270] ehci-platform 1c1c000.usb: EHCI Host Controller
[    5.358901] ehci-platform 1c1c000.usb: new USB bus registered, assigned bus number 2
[    5.367010] ehci-platform 1c1c000.usb: irq 34, io mem 0x01c1c000
[    5.403025] ehci-platform 1c1c000.usb: USB 2.0 started, EHCI 1.00
[    5.409398] usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
[    5.416221] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    5.423460] usb usb2: Product: EHCI Host Controller
[    5.428341] usb usb2: Manufacturer: Linux 4.14.15-sunxi ehci_hcd
[    5.434359] usb usb2: SerialNumber: 1c1c000.usb
[    5.444878] hub 2-0:1.0: USB hub found
[    5.448705] hub 2-0:1.0: 1 port detected
[    5.463202] ohci-platform 1c14400.usb: Generic Platform OHCI controller
[    5.469859] ohci-platform 1c14400.usb: new USB bus registered, assigned bus number 3
[    5.477918] ohci-platform 1c14400.usb: irq 31, io mem 0x01c14400
[    5.557589] usb usb3: New USB device found, idVendor=1d6b, idProduct=0001
[    5.564451] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    5.571680] usb usb3: Product: Generic Platform OHCI controller
[    5.575031] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[    5.583835] usb usb3: Manufacturer: Linux 4.14.15-sunxi ohci_hcd
[    5.589863] usb usb3: SerialNumber: 1c14400.usb
[    5.589878] ata1.00: ATA-10: ST4000LM024-2AN17V, 0001, max UDMA/133
[    5.589884] ata1.00: 7814037168 sectors, multi 0: LBA48 NCQ (depth 31/32)
[    5.608165] hub 3-0:1.0: USB hub found
[    5.611981] hub 3-0:1.0: 1 port detected
[    5.616872] ohci-platform 1c1c400.usb: Generic Platform OHCI controller
[    5.623598] ohci-platform 1c1c400.usb: new USB bus registered, assigned bus number 4
[    5.631624] ohci-platform 1c1c400.usb: irq 35, io mem 0x01c1c400
[    5.707745] usb usb4: New USB device found, idVendor=1d6b, idProduct=0001
[    5.714676] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    5.721954] usb usb4: Product: Generic Platform OHCI controller
[    5.727963] usb usb4: Manufacturer: Linux 4.14.15-sunxi ohci_hcd
[    5.734054] usb usb4: SerialNumber: 1c1c400.usb
[    5.746805] hub 4-0:1.0: USB hub found
[    5.750615] hub 4-0:1.0: 1 port detected
[    5.755727] usb_phy_generic usb_phy_generic.0.auto: usb_phy_generic.0.auto supply vcc not found, using dummy regulator
[    5.767076] musb-hdrc musb-hdrc.1.auto: MUSB HDRC host driver
[    5.772849] musb-hdrc musb-hdrc.1.auto: new USB bus registered, assigned bus number 5
[    5.781047] usb usb5: New USB device found, idVendor=1d6b, idProduct=0002
[    5.787903] usb usb5: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    5.795155] usb usb5: Product: MUSB HDRC host driver
[    5.800129] usb usb5: Manufacturer: Linux 4.14.15-sunxi musb-hcd
[    5.806170] usb usb5: SerialNumber: musb-hdrc.1.auto
[    5.811891] hub 5-0:1.0: USB hub found
[    5.815769] hub 5-0:1.0: 1 port detected
[    5.832503] of_cfs_init
[    5.835125] of_cfs_init: OK
[    5.838111] vcc3v0: disabling
[    5.840041] ata1.00: configured for UDMA/133
[    5.840770] scsi 0:0:0:0: Direct-Access     ATA      ST4000LM024-2AN1 0001 PQ: 0 ANSI: 5
[    5.841634] sd 0:0:0:0: Attached scsi generic sg0 type 0
[    5.842141] sd 0:0:0:0: [sda] 7814037168 512-byte logical blocks: (4.00 TB/3.64 TiB)
[    5.842148] sd 0:0:0:0: [sda] 4096-byte physical blocks
[    5.842210] sd 0:0:0:0: [sda] Write Protect is off
[    5.842217] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
[    5.842306] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    5.885690] vcc5v0: disabling
[    5.888678] vddio-csi1: disabling
[    5.892325] usb0-vbus: disabling
[    5.905456]  sda: sda1
[    5.911377] sd 0:0:0:0: [sda] Attached SCSI disk
[    5.924294] Freeing unused kernel memory: 1024K
[    6.004056] systemd-udevd[140]: starting version 215
[    6.120167] sunxi-mmc 1c11000.mmc: allocated mmc-pwrseq
[    6.123158] usb 3-1: new full-speed USB device number 2 using ohci-platform
[    6.183128] sunxi-mmc 1c11000.mmc: base:0xf101c000 irq:28
[    6.277175] mmc1: new DDR MMC card at address 0001
[    6.285756] mmcblk1: mmc1:0001 P1XXXX 3.60 GiB 
[    6.290895] mmcblk1boot0: mmc1:0001 P1XXXX partition 1 16.0 MiB
[    6.297649] mmcblk1boot1: mmc1:0001 P1XXXX partition 2 16.0 MiB
[    6.304808]  mmcblk1: p1
[    6.405184] usb 3-1: New USB device found, idVendor=0403, idProduct=6015
[    6.412110] usb 3-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[    6.419477] usb 3-1: Product: FT231X USB UART
[    6.424082] usb 3-1: Manufacturer: FTDI
[    6.428041] usb 3-1: SerialNumber: DN018W7V
[    9.348004] EXT4-fs (sda1): mounted filesystem with writeback data mode. Opts: (null)
[   10.398725] systemd[1]: systemd 215 running in system mode. (+PAM +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ -SECCOMP -APPARMOR)
[   10.412542] systemd[1]: Detected architecture 'arm'.
[   10.541809] systemd[1]: Set hostname to <lime2>.
[   12.644366] systemd[1]: Cannot add dependency job for unit display-manager.service, ignoring: Unit display-manager.service failed to load: No such file or directory.
[   12.663903] systemd[1]: Expecting device dev-ttyS0.device...
[   12.680433] systemd[1]: Starting Forward Password Requests to Wall Directory Watch.
[   12.688580] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
[   12.696363] systemd[1]: Starting Remote File Systems (Pre).
[   12.702351] systemd[1]: Reached target Remote File Systems (Pre).
[   12.708828] systemd[1]: Starting Arbitrary Executable File Formats File System Automount Point.
[   13.207363] fuse init (API version 7.26)
[   13.397347] Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
[   13.572762] systemd-udevd[232]: starting version 215
[   14.722137] sun4i-ss 1c15000.crypto-engine: Die ID 0
[   14.862608] usbcore: registered new interface driver usbserial
[   14.868722] usbcore: registered new interface driver usbserial_generic
[   14.875684] usbserial: USB Serial support registered for generic
[   14.940131] EXT4-fs (sda1): re-mounted. Opts: commit=600,errors=remount-ro
[   15.010305] usbcore: registered new interface driver ftdi_sio
[   15.016269] usbserial: USB Serial support registered for FTDI USB Serial Device
[   15.024084] ftdi_sio 3-1:1.0: FTDI USB Serial Device converter detected
[   15.031379] usb 3-1: Detected FT-X
[   15.059946] at24 1-0050: 2048 byte 24c16 EEPROM, writable, 16 bytes/write
[   15.169264] usb 3-1: FTDI USB Serial Device converter now attached to ttyUSB0
[   15.524996] random: crng init done
[   15.547002] Adding 131068k swap on /var/swap.  Priority:-2 extents:2 across:139260k FS
[   15.620089] EXT4-fs (mmcblk0p1): recovery complete
[   15.627993] EXT4-fs (mmcblk0p1): mounted filesystem with writeback data mode. Opts: (null)
[   16.669524] systemd-journald[244]: Received request to flush runtime journal from PID 1
[   17.123148] thermal thermal_zone0: failed to read out thermal zone (-110)
[   18.413470] RTL8211B Gigabit Ethernet stmmac-0:01: attached PHY driver [RTL8211B Gigabit Ethernet] (mii_bus:phy_addr=stmmac-0:01, irq=POLL)
[   18.427802] sun7i-dwmac 1c50000.ethernet eth0: RX IPC Checksum Offload disabled
[   18.435210] sun7i-dwmac 1c50000.ethernet eth0: No MAC Management Counters available
[   18.442879] sun7i-dwmac 1c50000.ethernet eth0: PTP not supported by HW
[   18.449784] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[   19.523959] sun7i-dwmac 1c50000.ethernet eth0: Link is Up - 1Gbps/Full - flow control rx/tx
[   19.532404] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[19229.640294] thermal thermal_zone0: failed to read out thermal zone (-110)
[19231.800287] thermal thermal_zone0: failed to read out thermal zone (-110)
[19233.960274] thermal thermal_zone0: failed to read out thermal zone (-110)
[19236.120261] thermal thermal_zone0: failed to read out thermal zone (-110)
[19238.280247] thermal thermal_zone0: failed to read out thermal zone (-110)
[19240.440232] thermal thermal_zone0: failed to read out thermal zone (-110)
[19242.600255] thermal thermal_zone0: failed to read out thermal zone (-110)
[19244.760203] thermal thermal_zone0: failed to read out thermal zone (-110)
[19246.920200] thermal thermal_zone0: failed to read out thermal zone (-110)
[19249.080193] thermal thermal_zone0: failed to read out thermal zone (-110)
[19249.650082] INFO: rcu_sched detected stalls on CPUs/tasks:
[19249.655608] 	0-...: (1 GPs behind) idle=116/140000000000000/0 softirq=211124/211128 fqs=1014 
[19249.664131] 	(detected by 1, t=2102 jiffies, g=119264, c=119263, q=101)
[19249.670755] Sending NMI from CPU 1 to CPUs 0:
[19259.900121] ata1.00: exception Emask 0x0 SAct 0x10000 SErr 0x0 action 0x6 frozen
[19259.907555] ata1.00: failed command: READ FPDMA QUEUED
[19259.912746] ata1.00: cmd 60/20:80:e0:02:0a/00:00:09:00:00/40 tag 16 ncq dma 16384 in
         res 40/00:00:00:4f:c2/00:00:00:00:00/40 Emask 0x4 (timeout)
[19259.927925] ata1.00: status: { DRDY }
[19259.931622] ata1: hard resetting link
[19260.261407] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[19260.840088] thermal thermal_zone0: failed to read out thermal zone (-110)
[19263.000097] thermal thermal_zone0: failed to read out thermal zone (-110)
[19265.160081] thermal thermal_zone0: failed to read out thermal zone (-110)
[19265.320044] ata1.00: qc timeout (cmd 0xec)
[19265.324172] ata1.00: failed to IDENTIFY (I/O error, err_mask=0x4)
[19265.330302] ata1.00: revalidation failed (errno=-5)
[19265.335195] ata1: hard resetting link
[19265.661344] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[19267.320065] thermal thermal_zone0: failed to read out thermal zone (-110)
[19269.480191] thermal thermal_zone0: failed to read out thermal zone (-110)
[19271.640031] thermal thermal_zone0: failed to read out thermal zone (-110)
[19273.800051] thermal thermal_zone0: failed to read out thermal zone (-110)
[19275.058307] systemd[1]: Starting Journal Service...
[19275.879982] ata1.00: qc timeout (cmd 0xec)
[19275.884118] ata1.00: failed to IDENTIFY (I/O error, err_mask=0x4)
[19275.890249] ata1.00: revalidation failed (errno=-5)
[19275.895136] ata1: limiting SATA link speed to 1.5 Gbps
[19275.900300] ata1: hard resetting link
[19275.960001] thermal thermal_zone0: failed to read out thermal zone (-110)
[19276.231277] ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
[19278.120001] thermal thermal_zone0: failed to read out thermal zone (-110)
[19280.279979] thermal thermal_zone0: failed to read out thermal zone (-110)
[19282.439970] thermal thermal_zone0: failed to read out thermal zone (-110)
[19284.599957] thermal thermal_zone0: failed to read out thermal zone (-110)
[19286.759938] thermal thermal_zone0: failed to read out thermal zone (-110)
[19288.919934] thermal thermal_zone0: failed to read out thermal zone (-110)
[19291.079917] thermal thermal_zone0: failed to read out thermal zone (-110)
[19293.239904] thermal thermal_zone0: failed to read out thermal zone (-110)
[19295.399901] thermal thermal_zone0: failed to read out thermal zone (-110)
[19297.559875] thermal thermal_zone0: failed to read out thermal zone (-110)
[19299.719863] thermal thermal_zone0: failed to read out thermal zone (-110)
[19301.879854] thermal thermal_zone0: failed to read out thermal zone (-110)
[19304.039836] thermal thermal_zone0: failed to read out thermal zone (-110)
[19306.199822] thermal thermal_zone0: failed to read out thermal zone (-110)
[19306.599785] ata1.00: qc timeout (cmd 0xec)
[19306.603916] ata1.00: failed to IDENTIFY (I/O error, err_mask=0x4)
[19306.610047] ata1.00: revalidation failed (errno=-5)
[19306.614929] ata1.00: disabled
[19306.617950] ata1: hard resetting link
[19306.951139] ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
[19306.957402] ata1: EH complete
[19306.960529] sd 0:0:0:0: [sda] tag#17 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[19306.968824] sd 0:0:0:0: [sda] tag#17 CDB: opcode=0x88 88 00 00 00 00 00 09 0a 02 e0 00 00 00 20 00 00
[19306.978068] print_req_error: I/O error, dev sda, sector 151651040
[19306.984514] sd 0:0:0:0: [sda] tag#18 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[19306.992833] sd 0:0:0:0: [sda] tag#18 CDB: opcode=0x88 88 00 00 00 00 00 09 0a 02 e8 00 00 00 08 00 00
[19307.002069] print_req_error: I/O error, dev sda, sector 151651048
[19307.009822] sd 0:0:0:0: [sda] tag#19 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[19307.018119] sd 0:0:0:0: [sda] tag#19 CDB: opcode=0x88 88 00 00 00 00 00 0d 9d 3f 58 00 00 00 08 00 00
[19307.027370] print_req_error: I/O error, dev sda, sector 228409176
[19307.033557] sd 0:0:0:0: [sda] tag#20 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[19307.041863] sd 0:0:0:0: [sda] tag#20 CDB: opcode=0x88 88 00 00 00 00 00 0d 9d 3f c0 00 00 00 08 00 00
[19307.051099] print_req_error: I/O error, dev sda, sector 228409280
[19307.057556] sd 0:0:0:0: [sda] tag#21 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[19307.065873] sd 0:0:0:0: [sda] tag#21 CDB: opcode=0x88 88 00 00 00 00 00 0d 9d 3f 58 00 00 00 08 00 00
[19307.075110] print_req_error: I/O error, dev sda, sector 228409176
[19307.086488] systemd[1]: systemd-journald.service: main process exited, code=killed, status=7/BUS
[19307.096874] systemd[1]: Unit systemd-journald.service entered failed state.
[19307.108866] systemd[1]: systemd-journald.service has no holdoff time, scheduling restart.
[19307.121960] systemd[1]: rsyslog.service: main process exited, code=killed, status=7/BUS
[19307.134009] systemd[1]: Unit rsyslog.service entered failed state.
[19307.159615] systemd[1]: Stopping Journal Service...
[19307.165033] systemd[1]: Starting Journal Service...
[19307.174569] systemd[1]: Started Journal Service.
[19307.185739] sd 0:0:0:0: [sda] tag#22 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[19307.194116] sd 0:0:0:0: [sda] tag#22 CDB: opcode=0x88 88 00 00 00 00 00 0d 9d 3f c0 00 00 00 08 00 00
[19307.203361] print_req_error: I/O error, dev sda, sector 228409280
[19307.345665] systemd[1]: systemd-journald.service has no holdoff time, scheduling restart.
[19307.354544] systemd[1]: rsyslog.service holdoff time over, scheduling restart.
[19307.383925] sd 0:0:0:0: [sda] tag#23 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[19307.392314] sd 0:0:0:0: [sda] tag#23 CDB: opcode=0x88 88 00 00 00 00 00 09 0a 44 10 00 00 00 20 00 00
[19307.401567] print_req_error: I/O error, dev sda, sector 151667728
[19307.419843] sd 0:0:0:0: [sda] tag#24 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[19307.428146] sd 0:0:0:0: [sda] tag#24 CDB: opcode=0x88 88 00 00 00 00 00 0d 9d 3f c0 00 00 00 08 00 00
[19307.437395] print_req_error: I/O error, dev sda, sector 228409280
[19307.448606] sd 0:0:0:0: [sda] tag#25 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[19307.456980] sd 0:0:0:0: [sda] tag#25 CDB: opcode=0x88 88 00 00 00 00 00 09 0a 44 10 00 00 00 08 00 00
[19307.466218] print_req_error: I/O error, dev sda, sector 151667728
[19307.543581] sd 0:0:0:0: [sda] tag#26 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[19307.551956] sd 0:0:0:0: [sda] tag#26 CDB: opcode=0x88 88 00 00 00 00 00 0d 9d 3f c0 00 00 00 08 00 00
[19307.561199] print_req_error: I/O error, dev sda, sector 228409280
[19308.068643] systemd[7314]: Failed at step EXEC spawning /usr/sbin/rsyslogd: Input/output error
[19308.318928] systemd[7316]: Failed at step EXEC spawning /usr/sbin/rsyslogd: Input/output error
[19308.359849] thermal thermal_zone0: failed to read out thermal zone (-110)
[19308.568789] systemd[7318]: Failed at step EXEC spawning /usr/sbin/rsyslogd: Input/output error
[19310.519807] thermal thermal_zone0: failed to read out thermal zone (-110)
[19312.680555] thermal thermal_zone0: failed to read out thermal zone (-110)
[19312.699686] INFO: rcu_sched detected stalls on CPUs/tasks:
[19312.705188] 	0-...: (1 GPs behind) idle=116/140000000000000/0 softirq=211124/211128 fqs=3551 
[19312.713711] 	(detected by 1, t=8407 jiffies, g=119264, c=119263, q=3474)
[19312.720421] Sending NMI from CPU 1 to CPUs 0:
[19323.879702] thermal thermal_zone0: failed to read out thermal zone (-110)
[19326.039691] thermal thermal_zone0: failed to read out thermal zone (-110)
[19328.199684] thermal thermal_zone0: failed to read out thermal zone (-110)
[19330.359671] thermal thermal_zone0: failed to read out thermal zone (-110)
[19332.519654] thermal thermal_zone0: failed to read out thermal zone (-110)
[19334.679644] thermal thermal_zone0: failed to read out thermal zone (-110)
[19336.839654] thermal thermal_zone0: failed to read out thermal zone (-110)
[19338.999618] thermal thermal_zone0: failed to read out thermal zone (-110)
[19341.159594] thermal thermal_zone0: failed to read out thermal zone (-110)
[19343.319587] thermal thermal_zone0: failed to read out thermal zone (-110)
[19345.479573] thermal thermal_zone0: failed to read out thermal zone (-110)
[19347.639566] thermal thermal_zone0: failed to read out thermal zone (-110)
[19349.731586] systemd[1]: systemd-logind.service stop-sigterm timed out. Killing.
[19349.799550] thermal thermal_zone0: failed to read out thermal zone (-110)
[19351.959542] thermal thermal_zone0: failed to read out thermal zone (-110)
[19354.119522] thermal thermal_zone0: failed to read out thermal zone (-110)
[19356.279502] thermal thermal_zone0: failed to read out thermal zone (-110)
[19358.439499] thermal thermal_zone0: failed to read out thermal zone (-110)
[19360.599479] thermal thermal_zone0: failed to read out thermal zone (-110)
[19362.759476] thermal thermal_zone0: failed to read out thermal zone (-110)
[19364.919499] thermal thermal_zone0: failed to read out thermal zone (-110)
[19367.079435] thermal thermal_zone0: failed to read out thermal zone (-110)
[19369.239427] thermal thermal_zone0: failed to read out thermal zone (-110)
[19371.399444] thermal thermal_zone0: failed to read out thermal zone (-110)
[19373.559402] thermal thermal_zone0: failed to read out thermal zone (-110)
[19375.719385] thermal thermal_zone0: failed to read out thermal zone (-110)
[19375.749293] INFO: rcu_sched detected stalls on CPUs/tasks:
[19375.754811] 	0-...: (1 GPs behind) idle=116/140000000000000/0 softirq=211124/211128 fqs=6102 
[19375.763334] 	(detected by 1, t=14712 jiffies, g=119264, c=119263, q=3721)
[19375.770132] Sending NMI from CPU 1 to CPUs 0:

 

[&ergio]

I confirm this bug. After updating orange pi zero to 4.14.15-sunxi it hangs occasionally.

 

Feb  4 06:32:27 localhost kernel: [24810.496488] NMI backtrace for cpu 0
Feb  4 06:32:27 localhost kernel: [24810.496498] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.14.15-sunxi #28
Feb  4 06:32:27 localhost kernel: [24810.496501] Hardware name: Allwinner sun8i Family
Feb  4 06:32:27 localhost kernel: [24810.496506] task: c0d07780 task.stack: c0d00000
Feb  4 06:32:27 localhost kernel: [24810.496523] PC is at arch_cpu_idle+0x28/0x2c
Feb  4 06:32:27 localhost kernel: [24810.496529] LR is at arch_cpu_idle+0x27/0x2c
Feb  4 06:32:27 localhost kernel: [24810.496533] pc : [<c01070e8>]    lr : [<c01070e7>]    psr: 40070033
Feb  4 06:32:27 localhost kernel: [24810.496536] sp : c0d01f80  ip : 00000018  fp : c0c5ea30
Feb  4 06:32:27 localhost kernel: [24810.496539] r10: 00000000  r9 : 00000000  r8 : c0ddd8eb
Feb  4 06:32:27 localhost kernel: [24810.496544] r7 : c0cb6438  r6 : c0d03f6c  r5 : c0d03fcc  r4 : ffffe000
Feb  4 06:32:27 localhost kernel: [24810.496547] r3 : c0116561  r2 : 00000000  r1 : 00000000  r0 : 00000001
Feb  4 06:32:27 localhost kernel: [24810.496554] Flags: nZcv  IRQs on  FIQs on  Mode SVC_32  ISA Thumb  Segment none
Feb  4 06:32:27 localhost kernel: [24810.496558] Control: 50c5387d  Table: 5a21006a  DAC: 00000051
Feb  4 06:32:27 localhost kernel: [24810.496564] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.14.15-sunxi #28
Feb  4 06:32:27 localhost kernel: [24810.496566] Hardware name: Allwinner sun8i Family
Feb  4 06:32:27 localhost kernel: [24810.496592] [<c010db15>] (unwind_backtrace) from [<c010a0d9>] (show_stack+0x11/0x1
4)
Feb  4 06:32:27 localhost kernel: [24810.496605] [<c010a0d9>] (show_stack) from [<c0867e49>] (dump_stack+0x69/0x78)
Feb  4 06:32:27 localhost kernel: [24810.496621] [<c0867e49>] (dump_stack) from [<c086bc0f>] (nmi_cpu_backtrace+0x8b/0x
d4)
Feb  4 06:32:27 localhost kernel: [24810.496634] [<c086bc0f>] (nmi_cpu_backtrace) from [<c010ca01>] (handle_IPI+0x75/0x
278)
Feb  4 06:32:27 localhost kernel: [24810.496645] [<c010ca01>] (handle_IPI) from [<c01013e3>] (gic_handle_irq+0x67/0x68)
Feb  4 06:32:27 localhost kernel: [24810.496654] [<c01013e3>] (gic_handle_irq) from [<c010aa25>] (__irq_svc+0x65/0x94)
Feb  4 06:32:27 localhost kernel: [24810.496659] Exception stack(0xc0d01f30 to 0xc0d01f78)
Feb  4 06:32:27 localhost kernel: [24810.496666] 1f20:                                     00000001 00000000 00000000 c
0116561
Feb  4 06:32:27 localhost kernel: [24810.496675] 1f40: ffffe000 c0d03fcc c0d03f6c c0cb6438 c0ddd8eb 00000000 00000000 c
0c5ea30
Feb  4 06:32:27 localhost kernel: [24810.496682] 1f60: 00000018 c0d01f80 c01070e7 c01070e8 40070033 ffffffff
Feb  4 06:32:27 localhost kernel: [24810.496693] [<c010aa25>] (__irq_svc) from [<c01070e8>] (arch_cpu_idle+0x28/0x2c)
Feb  4 06:32:27 localhost kernel: [24810.496708] [<c01070e8>] (arch_cpu_idle) from [<c014c93d>] (do_idle+0x115/0x16c)
Feb  4 06:32:27 localhost kernel: [24810.496721] [<c014c93d>] (do_idle) from [<c014cb89>] (cpu_startup_entry+0x19/0x1c)
Feb  4 06:32:27 localhost kernel: [24810.496733] [<c014cb89>] (cpu_startup_entry) from [<c0c00ad5>] (start_kernel+0x353
/0x36a)
Feb  4 06:32:27 localhost kernel: [24810.497496] rcu_sched kthread starved for 6304 jiffies! g12210 c12209 f0x0 RCU_GP_
WAIT_FQS(3) ->state=0x402 ->cpu=0
Feb  4 06:32:27 localhost kernel: [24810.497502] rcu_sched       I    0     8      2 0x00000000
Feb  4 06:32:27 localhost kernel: [24810.497523] [<c08762d9>] (__schedule) from [<c087684f>] (schedule+0x2f/0x68)
Feb  4 06:32:27 localhost kernel: [24810.497538] [<c087684f>] (schedule) from [<c08790ed>] (schedule_timeout+0x75/0x314
)
Feb  4 06:32:27 localhost kernel: [24810.497555] [<c08790ed>] (schedule_timeout) from [<c0164f95>] (rcu_gp_kthread+0x41
5/0x674)
Feb  4 06:32:27 localhost kernel: [24810.497572] [<c0164f95>] (rcu_gp_kthread) from [<c0131a9d>] (kthread+0xfd/0x104)
Feb  4 06:32:27 localhost kernel: [24810.497587] [<c0131a9d>] (kthread) from [<c0106719>] (ret_from_fork+0x11/0x38)
Feb  4 06:32:50 localhost kernel: [24833.077001] INFO: rcu_sched detected stalls on CPUs/tasks:
Feb  4 06:32:50 localhost kernel: [24833.077650]        0-...: (0 ticks this GP) idle=252/140000000000000/0 softirq=212
452/212452 fqs=0
Feb  4 06:32:50 localhost kernel: [24833.077743]        (detected by 3, t=2102 jiffies, g=12216, c=12215, q=2956)
Feb  4 06:32:50 localhost kernel: [24833.077921] Sending NMI from CPU 3 to CPUs 0:

[Wladimir Mutel]

I had similar experience on my Cubietruck A20 (rcu_sched stalls and rejected forks)

It happened after my first reboot with kernel 4.14.15-sunxi (after some uptime with kernel 4.13.16)
I just run 'sudo cruft' at that time
 

Feb  2 15:58:35 localhost kernel: [102644.225181] INFO: rcu_sched detected stalls on CPUs/tasks:
Feb  2 15:58:35 localhost kernel: [102644.225228]       1-...: (1 GPs behind) idle=272/140000000000000/0 softirq=1869227/1869228 fqs=1050
Feb  2 15:58:35 localhost kernel: [102644.225232]       (detected by 0, t=2102 jiffies, g=798178, c=798177, q=5594)
Feb  2 15:58:35 localhost kernel: [102644.225258] Sending NMI from CPU 0 to CPUs 1:
Feb  2 15:59:38 localhost kernel: [102707.275047] INFO: rcu_sched detected stalls on CPUs/tasks:
Feb  2 15:59:38 localhost kernel: [102707.275093]       1-...: (1 GPs behind) idle=272/140000000000000/0 softirq=1869227/1869228 fqs=3702
Feb  2 15:59:38 localhost kernel: [102707.275097]       (detected by 0, t=8407 jiffies, g=798178, c=798177, q=68757)
Feb  2 15:59:38 localhost kernel: [102707.275122] Sending NMI from CPU 0 to CPUs 1:
Feb  2 15:59:51 localhost kernel: [102730.056354] cgroup: fork rejected by pids controller in /user.slice/user-1001.slice/session-222.scope
Feb  2 16:00:41 localhost kernel: [102770.324854] INFO: rcu_sched detected stalls on CPUs/tasks:
Feb  2 16:00:41 localhost kernel: [102770.324890]       1-...: (1 GPs behind) idle=272/140000000000000/0 softirq=1869227/1869228 fqs=6354
Feb  2 16:00:41 localhost kernel: [102770.324894]       (detected by 0, t=14712 jiffies, g=798178, c=798177, q=89964)
Feb  2 16:00:41 localhost kernel: [102770.324918] Sending NMI from CPU 0 to CPUs 1:
Feb  2 16:01:45 localhost kernel: [102833.374711] INFO: rcu_sched detected stalls on CPUs/tasks:
Feb  2 16:01:45 localhost kernel: [102833.374751]       1-...: (1 GPs behind) idle=272/140000000000000/0 softirq=1869227/1869228 fqs=9006
Feb  2 16:01:45 localhost kernel: [102833.374756]       (detected by 0, t=21017 jiffies, g=798178, c=798177, q=90505)
Feb  2 16:01:45 localhost kernel: [102833.374781] Sending NMI from CPU 0 to CPUs 1:
Feb  2 16:02:48 localhost kernel: [102896.424568] INFO: rcu_sched detected stalls on CPUs/tasks:
Feb  2 16:02:48 localhost kernel: [102896.424608]       1-...: (1 GPs behind) idle=272/140000000000000/0 softirq=1869227/1869228 fqs=11657
Feb  2 16:02:48 localhost kernel: [102896.424612]       (detected by 0, t=27322 jiffies, g=798178, c=798177, q=91242)
Feb  2 16:02:48 localhost kernel: [102896.424636] Sending NMI from CPU 0 to CPUs 1:
Feb  2 16:03:51 localhost kernel: [102959.474426] INFO: rcu_sched detected stalls on CPUs/tasks:
Feb  2 16:03:51 localhost kernel: [102959.474465]       1-...: (1 GPs behind) idle=272/140000000000000/0 softirq=1869227/1869228 fqs=14307
Feb  2 16:03:51 localhost kernel: [102959.474469]       (detected by 0, t=33627 jiffies, g=798178, c=798177, q=92168)
Feb  2 16:03:51 localhost kernel: [102959.474493] Sending NMI from CPU 0 to CPUs 1:
Feb  2 16:04:31 localhost kernel: [103009.885942] cgroup: fork rejected by pids controller in /user.slice/user-1001.slice/session-229.scope

Then I rebooted the board locally by its power switch, and nothing similar happened after that

Edited February 7, 2018 by Wladimir Mutel

[René Kliment]

The only workaround I have been able to come up so far is capping the CPU frequency pretty low - 480 MHz max - bigger wouldn't work for me and would always stall after a couple of hours.
I've been able to run it for 2 days straight now with 

cpufreq-set --max 500Mhz

 

[&ergio]

Workaround works!

[Igor]

It seems that latest nightly build (4.14.18-sunxi/4.14.18-sunxi64) also works just fine. I am currently heavily stressing out:

- Cubietruck
- Orange Prime
- Orange 2E

Uptime: 15 hours 56 minutes 50 seconds

If they are still up by the end of the day, I'll push kernel to the repository.

[René Kliment]

10 hours ago, Igor said:

It seems that latest nightly build (4.14.18-sunxi/4.14.18-sunxi64) also works just fine. I am currently heavily stressing out:

- Cubietruck
- Orange Prime
- Orange 2E

Uptime: 15 hours 56 minutes 50 seconds

If they are still up by the end of the day, I'll push kernel to the repository.

Yes, it seems that .17 and .18 work fine. Thanks!

[jonik]

Same crashes with Orange pi pc 

Linux orangepipc 4.14.18-sunxi #24 SMP Fri Feb 9 16:24:32 CET 2018 armv7l GNU/Linux
made a cpufreq-set --max 500Mhz

I will see for how long it wont crash

 

serial_console.log

[René Kliment]

12 hours ago, jonik said:

Same crashes with Orange pi pc 

Linux orangepipc 4.14.18-sunxi #24 SMP Fri Feb 9 16:24:32 CET 2018 armv7l GNU/Linux
made a cpufreq-set --max 500Mhz

I will see for how long it wont crash

 

serial_console.log

 

Yes, I can confirm I had freezes on my OPi +2e with a few latest 4.14.x kernels and I had to cap the CPU freq to 648 MHz - anything above would lead to a system freeze. I don't have a log unfortunately, but I believe it would be same as yours.

I see there is kernel 4.17.6 available, testing it now to see if it is fixed or not.
Maybe a patch that fixed this before got left out?
@Igor

[Igor]

1 hour ago, René Kliment said:

Maybe a patch that fixed this before got left out?

It's also possible that some boards are much lower quality than expected with our already conservative settings. It's not just kernel, but also u-boot. Both are changed now.

[René Kliment]

On 7/18/2018 at 10:40 AM, Igor said:

It's also possible that some boards are much lower quality than expected with our already conservative settings. It's not just kernel, but also u-boot. Both are changed now.

 

Board quality is definitely a factor. However, it worked fine for me in early 4.14 kernels, then got broken, then got fixed again and now it's broken again so it seems like a kernel and/or uboot issue. Let's hope it will magically fix itself again :-O

I can confirm that it froze with 4.17.6. I've upgraded to 4.17.8 and connected the serial console to catch the dmesg output when it freezes again.

[Links2004]

 

 

Hi,

 

can confirm that this still is a big Issue, (running a Orange PI PC)

since I updatet from 4.14.18-sunxi to 4.17.7-sunxi the system is in a dead state very.

before the update the system where running for month stable.

 

Interesting is that the first notable think is that ssh stops working (aktive session is dead and new session gets a timeout),

ping and the home automation stuff running there will run ~1hour more and then the system is dead complete.

 

some log which may or may not helps here:

 

Spoiler

Jul 21 17:00:01 localhost CRON[2647]: (CRON) info (No MTA installed, discarding output)
Jul 21 17:05:01 localhost CRON[2719]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Jul 21 17:09:16 localhost systemd[1]: Started Session c5 of user root.
Jul 21 17:09:48 localhost kernel: [ 4588.342035] hrtimer: interrupt took 1017921 ns
Jul 21 17:15:01 localhost CRON[2897]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Jul 21 17:17:02 localhost CRON[2916]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul 21 17:25:01 localhost CRON[2949]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Jul 21 17:35:01 localhost CRON[3013]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Jul 21 17:44:17 localhost kernel: [ 6656.904339] INFO: rcu_sched self-detected stall on CPU
Jul 21 17:44:17 localhost kernel: [ 6656.904365]     3-...!: (1 ticks this GP) idle=70a/0/1 softirq=349837/349837 fqs=0
Jul 21 17:44:17 localhost kernel: [ 6656.904368]      (t=1022204 jiffies g=255720 c=255719 q=0)
Jul 21 17:44:17 localhost kernel: [ 6656.904384] rcu_sched kthread starved for 1022204 jiffies! g255720 c255719 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x402 ->cpu=3
Jul 21 17:44:17 localhost kernel: [ 6656.904388] RCU grace-period kthread stack dump:
Jul 21 17:44:17 localhost kernel: [ 6656.904392] rcu_sched       I    0     9      2 0x00000000
Jul 21 17:44:17 localhost kernel: [ 6656.904430] [<c080e9c7>] (__schedule) from [<c080ef2b>] (schedule+0x2f/0x68)
Jul 21 17:44:17 localhost kernel: [ 6656.904446] [<c080ef2b>] (schedule) from [<c08114e1>] (schedule_timeout+0x69/0x2e4)
Jul 21 17:44:17 localhost kernel: [ 6656.904463] [<c08114e1>] (schedule_timeout) from [<c0165869>] (rcu_gp_kthread+0x405/0x638)
Jul 21 17:44:17 localhost kernel: [ 6656.904478] [<c0165869>] (rcu_gp_kthread) from [<c0130569>] (kthread+0xfd/0x104)
Jul 21 17:44:17 localhost kernel: [ 6656.904492] [<c0130569>] (kthread) from [<c01010f9>] (ret_from_fork+0x11/0x38)
Jul 21 17:44:17 localhost kernel: [ 6656.904497] Exception stack(0xef117fb0 to 0xef117ff8)
Jul 21 17:44:17 localhost kernel: [ 6656.904505] 7fa0:                                     00000000 00000000 00000000 00000000
Jul 21 17:44:17 localhost kernel: [ 6656.904514] 7fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul 21 17:44:17 localhost kernel: [ 6656.904522] 7fe0: 00000000 00000000 00000000 00000000 00000013 00000000
Jul 21 17:44:17 localhost kernel: [ 6656.904539] Sending NMI from CPU 3 to CPUs 1:
Jul 21 17:44:17 localhost kernel: [ 6656.904548] NMI backtrace for cpu 1
Jul 21 17:44:17 localhost kernel: [ 6656.904558] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.17.7-sunxi #125
Jul 21 17:44:17 localhost kernel: [ 6656.904561] Hardware name: Allwinner sun8i Family
Jul 21 17:44:17 localhost kernel: [ 6656.904571] PC is at arch_cpu_idle+0x28/0x2c
Jul 21 17:44:17 localhost kernel: [ 6656.904577] LR is at arch_cpu_idle+0x27/0x2c
Jul 21 17:44:17 localhost kernel: [ 6656.904581] pc : [<c0107774>]    lr : [<c0107773>]    psr: 400f0033
Jul 21 17:44:17 localhost kernel: [ 6656.904585] sp : ef11dfb0  ip : 00000000  fp : c0dd199f
Jul 21 17:44:17 localhost kernel: [ 6656.904588] r10: 00000000  r9 : c0cad540  r8 : 00000000
Jul 21 17:44:17 localhost kernel: [ 6656.904593] r7 : 00000002  r6 : c0d04d78  r5 : c0d04d28  r4 : ffffe000
Jul 21 17:44:17 localhost kernel: [ 6656.904597] r3 : c0115981  r2 : ef6b2428  r1 : 00429840  r0 : 00000000
Jul 21 17:44:17 localhost kernel: [ 6656.904603] Flags: nZcv  IRQs on  FIQs on  Mode SVC_32  ISA Thumb  Segment none
Jul 21 17:44:17 localhost kernel: [ 6656.904608] Control: 50c5387d  Table: 6d0e406a  DAC: 00000051
Jul 21 17:44:17 localhost kernel: [ 6656.904614] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.17.7-sunxi #125
Jul 21 17:44:17 localhost kernel: [ 6656.904616] Hardware name: Allwinner sun8i Family
Jul 21 17:44:17 localhost kernel: [ 6656.904632] [<c010cef9>] (unwind_backtrace) from [<c010a471>] (show_stack+0x11/0x14)
Jul 21 17:44:17 localhost kernel: [ 6656.904645] [<c010a471>] (show_stack) from [<c08006b1>] (dump_stack+0x69/0x78)
Jul 21 17:44:17 localhost kernel: [ 6656.904656] [<c08006b1>] (dump_stack) from [<c08043b7>] (nmi_cpu_backtrace+0x8b/0xd4)
Jul 21 17:44:17 localhost kernel: [ 6656.904668] [<c08043b7>] (nmi_cpu_backtrace) from [<c010c127>] (handle_IPI+0x9f/0x244)
Jul 21 17:44:17 localhost kernel: [ 6656.904683] [<c010c127>] (handle_IPI) from [<c04dace7>] (gic_handle_irq+0x73/0x74)
Jul 21 17:44:17 localhost kernel: [ 6656.904695] [<c04dace7>] (gic_handle_irq) from [<c0101a25>] (__irq_svc+0x65/0x94)
Jul 21 17:44:17 localhost kernel: [ 6656.904699] Exception stack(0xef11df60 to 0xef11dfa8)
Jul 21 17:44:17 localhost kernel: [ 6656.904709] df60: 00000000 00429840 ef6b2428 c0115981 ffffe000 c0d04d28 c0d04d78 00000002
Jul 21 17:44:17 localhost kernel: [ 6656.904718] df80: 00000000 c0cad540 00000000 c0dd199f 00000000 ef11dfb0 c0107773 c0107774
Jul 21 17:44:17 localhost kernel: [ 6656.904722] dfa0: 400f0033 ffffffff
Jul 21 17:44:17 localhost kernel: [ 6656.904734] [<c0101a25>] (__irq_svc) from [<c0107774>] (arch_cpu_idle+0x28/0x2c)
Jul 21 17:44:17 localhost kernel: [ 6656.904747] [<c0107774>] (arch_cpu_idle) from [<c013c001>] (do_idle+0x131/0x1b0)
Jul 21 17:44:17 localhost kernel: [ 6656.904757] [<c013c001>] (do_idle) from [<c013c255>] (cpu_startup_entry+0x19/0x1c)
Jul 21 17:44:17 localhost kernel: [ 6656.904767] [<c013c255>] (cpu_startup_entry) from [<40102451>] (0x40102451)
Jul 21 17:44:57 localhost kernel: [ 6656.905551] Sending NMI from CPU 3 to CPUs 2:
Jul 21 17:44:57 localhost kernel: [ 6656.905561] NMI backtrace for cpu 2
Jul 21 17:44:57 localhost kernel: [ 6656.905569] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.17.7-sunxi #125
Jul 21 17:44:57 localhost kernel: [ 6656.905572] Hardware name: Allwinner sun8i Family
Jul 21 17:44:57 localhost kernel: [ 6656.905582] PC is at arch_cpu_idle+0x28/0x2c
Jul 21 17:44:57 localhost kernel: [ 6656.905588] LR is at arch_cpu_idle+0x27/0x2c
Jul 21 17:44:57 localhost kernel: [ 6656.905592] pc : [<c0107774>]    lr : [<c0107773>]    psr: 40070033
Jul 21 17:44:57 localhost kernel: [ 6656.905596] sp : ef11ffb0  ip : 00000000  fp : c0dd199f
Jul 21 17:44:57 localhost kernel: [ 6656.905599] r10: 00000000  r9 : c0cad540  r8 : 00000000
Jul 21 17:44:57 localhost kernel: [ 6656.905603] r7 : 00000004  r6 : c0d04d78  r5 : c0d04d28  r4 : ffffe000
Jul 21 17:44:57 localhost kernel: [ 6656.905608] r3 : c0115981  r2 : ef6c3428  r1 : 0042db1c  r0 : 00000000
Jul 21 17:44:57 localhost kernel: [ 6656.905613] Flags: nZcv  IRQs on  FIQs on  Mode SVC_32  ISA Thumb  Segment none
Jul 21 17:44:57 localhost kernel: [ 6656.905617] Control: 50c5387d  Table: 6e35406a  DAC: 00000051
Jul 21 17:44:57 localhost kernel: [ 6656.905623] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.17.7-sunxi #125
Jul 21 17:44:57 localhost kernel: [ 6656.905625] Hardware name: Allwinner sun8i Family
Jul 21 17:44:57 localhost kernel: [ 6656.905641] [<c010cef9>] (unwind_backtrace) from [<c010a471>] (show_stack+0x11/0x14)
Jul 21 17:44:57 localhost kernel: [ 6656.905653] [<c010a471>] (show_stack) from [<c08006b1>] (dump_stack+0x69/0x78)
Jul 21 17:44:57 localhost kernel: [ 6656.905669] [<c08006b1>] (dump_stack) from [<c08043b7>] (nmi_cpu_backtrace+0x8b/0xd4)
Jul 21 17:44:57 localhost kernel: [ 6656.905681] [<c08043b7>] (nmi_cpu_backtrace) from [<c010c127>] (handle_IPI+0x9f/0x244)
Jul 21 17:44:57 localhost kernel: [ 6656.905699] [<c010c127>] (handle_IPI) from [<c04dace7>] (gic_handle_irq+0x73/0x74)
Jul 21 17:44:57 localhost kernel: [ 6656.905712] [<c04dace7>] (gic_handle_irq) from [<c0101a25>] (__irq_svc+0x65/0x94)
Jul 21 17:44:57 localhost kernel: [ 6656.905715] Exception stack(0xef11ff60 to 0xef11ffa8)
Jul 21 17:44:57 localhost kernel: [ 6656.905725] ff60: 00000000 0042db1c ef6c3428 c0115981 ffffe000 c0d04d28 c0d04d78 00000004
Jul 21 17:44:57 localhost kernel: [ 6656.905734] ff80: 00000000 c0cad540 00000000 c0dd199f 00000000 ef11ffb0 c0107773 c0107774
Jul 21 17:44:57 localhost kernel: [ 6656.905738] ffa0: 40070033 ffffffff
Jul 21 17:44:57 localhost kernel: [ 6656.905750] [<c0101a25>] (__irq_svc) from [<c0107774>] (arch_cpu_idle+0x28/0x2c)
Jul 21 17:44:57 localhost kernel: [ 6656.905762] [<c0107774>] (arch_cpu_idle) from [<c013c001>] (do_idle+0x131/0x1b0)
Jul 21 17:44:57 localhost kernel: [ 6656.905765] INFO: rcu_sched self-detected stall on CPU
Jul 21 17:44:57 localhost kernel: [ 6656.905774] [<c013c001>] (do_idle) from [<c013c255>] (cpu_startup_entry+0x19/0x1c)
Jul 21 17:44:57 localhost kernel: [ 6656.905783] [<c013c255>] (cpu_startup_entry) from [<40102451>] (0x40102451)
Jul 21 17:44:57 localhost kernel: [ 6656.905795] INFO: rcu_sched self-detected stall on CPU
Jul 21 17:44:57 localhost kernel: [ 6656.906560] NMI backtrace for cpu 3
Jul 21 17:44:57 localhost kernel: [ 6656.906569] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.17.7-sunxi #125
Jul 21 17:44:57 localhost kernel: [ 6656.906572] Hardware name: Allwinner sun8i Family
Jul 21 17:44:57 localhost kernel: [ 6656.906586] [<c010cef9>] (unwind_backtrace) from [<c010a471>] (show_stack+0x11/0x14)
Jul 21 17:44:57 localhost kernel: [ 6656.906598] [<c010a471>] (show_stack) from [<c08006b1>] (dump_stack+0x69/0x78)
Jul 21 17:44:57 localhost kernel: [ 6656.906609] [<c08006b1>] (dump_stack) from [<c08043ff>] (nmi_cpu_backtrace+0xd3/0xd4)
Jul 21 17:44:57 localhost kernel: [ 6656.906621] [<c08043ff>] (nmi_cpu_backtrace) from [<c080448d>] (nmi_trigger_cpumask_backtrace+0x8d/0xc4)
Jul 21 17:44:57 localhost kernel: [ 6656.906636] [<c080448d>] (nmi_trigger_cpumask_backtrace) from [<c016766d>] (rcu_dump_cpu_stacks+0x77/0x94)
Jul 21 17:44:57 localhost kernel: [ 6656.906651] [<c016766d>] (rcu_dump_cpu_stacks) from [<c0166df1>] (rcu_check_callbacks+0x461/0x630)
Jul 21 17:44:57 localhost kernel: [ 6656.906666] [<c0166df1>] (rcu_check_callbacks) from [<c016b063>] (update_process_times+0x2b/0x48)
Jul 21 17:44:57 localhost kernel: [ 6656.906682] [<c016b063>] (update_process_times) from [<c01784eb>] (tick_sched_timer+0x37/0x74)
Jul 21 17:44:57 localhost kernel: [ 6656.906697] [<c01784eb>] (tick_sched_timer) from [<c016b89b>] (__hrtimer_run_queues+0xff/0x214)
Jul 21 17:44:57 localhost kernel: [ 6656.906710] [<c016b89b>] (__hrtimer_run_queues) from [<c016c321>] (hrtimer_interrupt+0xb5/0x1fc)
Jul 21 17:44:57 localhost kernel: [ 6656.906727] [<c016c321>] (hrtimer_interrupt) from [<c06cc119>] (arch_timer_handler_phys+0x25/0x28)
Jul 21 17:44:57 localhost kernel: [ 6656.906745] [<c06cc119>] (arch_timer_handler_phys) from [<c015ea3f>] (handle_percpu_devid_irq+0x5f/0x19c)
Jul 21 17:44:57 localhost kernel: [ 6656.906759] [<c015ea3f>] (handle_percpu_devid_irq) from [<c015b13d>] (generic_handle_irq+0x1d/0x28)
Jul 21 17:44:57 localhost kernel: [ 6656.906772] [<c015b13d>] (generic_handle_irq) from [<c015b5dd>] (__handle_domain_irq+0x45/0x84)
Jul 21 17:44:57 localhost kernel: [ 6656.906784] [<c015b5dd>] (__handle_domain_irq) from [<c04dacb7>] (gic_handle_irq+0x43/0x74)
Jul 21 17:44:57 localhost kernel: [ 6656.906796] [<c04dacb7>] (gic_handle_irq) from [<c0101a25>] (__irq_svc+0x65/0x94)
Jul 21 17:44:57 localhost kernel: [ 6656.906802] Exception stack(0xef121f60 to 0xef121fa8)
Jul 21 17:44:57 localhost kernel: [ 6656.906812] 1f60: 00000000 00430708 ef6d4428 c0115981 ffffe000 c0d04d28 c0d04d78 00000008
Jul 21 17:44:57 localhost kernel: [ 6656.906822] 1f80: 00000000 c0cad540 00000000 c0dd199f 00000000 ef121fb0 c0107773 c0107774
Jul 21 17:44:57 localhost kernel: [ 6656.906828] 1fa0: 400f0033 ffffffff
Jul 21 17:44:57 localhost kernel: [ 6656.906840] [<c0101a25>] (__irq_svc) from [<c0107774>] (arch_cpu_idle+0x28/0x2c)
Jul 21 17:44:57 localhost kernel: [ 6656.906851] [<c0107774>] (arch_cpu_idle) from [<c013c001>] (do_idle+0x131/0x1b0)
Jul 21 17:44:57 localhost kernel: [ 6656.906861] [<c013c001>] (do_idle) from [<c013c255>] (cpu_startup_entry+0x19/0x1c)
Jul 21 17:44:57 localhost kernel: [ 6656.906871] [<c013c255>] (cpu_startup_entry) from [<40102451>] (0x40102451)
Jul 21 17:44:57 localhost kernel: [ 6656.906887]     1-...!: (1 ticks this GP) idle=852/1/1073741826 softirq=285578/285578 fqs=0
Jul 21 17:44:57 localhost kernel: [ 6656.906889]
Jul 21 17:44:57 localhost kernel: [ 6656.906897]     2-...!: (4 GPs behind) idle=b22/0/1 softirq=352408/352410 fqs=0
Jul 21 17:44:57 localhost kernel: [ 6656.906899]
Jul 21 17:44:57 localhost kernel: [ 6656.906903]  (t=1022204 jiffies g=255720 c=255719 q=0)
Jul 21 17:44:57 localhost kernel: [ 6656.906912] rcu_sched kthread starved for 1022204 jiffies! g255720 c255719 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x0 ->cpu=3
Jul 21 17:44:57 localhost kernel: [ 6656.906915]  (t=1022204 jiffies g=255720 c=255719 q=0)
Jul 21 17:44:57 localhost kernel: [ 6656.906923] rcu_sched kthread starved for 1022204 jiffies! g255720 c255719 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x0 ->cpu=3
Jul 21 17:44:57 localhost kernel: [ 6656.906925] RCU grace-period kthread stack dump:
Jul 21 17:44:57 localhost kernel: [ 6656.906927] RCU grace-period kthread stack dump:
Jul 21 17:44:57 localhost kernel: [ 6656.906929] rcu_sched       R  running task
Jul 21 17:44:57 localhost kernel: [ 6656.906933] rcu_sched       R  running task
Jul 21 17:44:57 localhost kernel: [ 6656.906939]     0     9      2 0x00000000
Jul 21 17:44:57 localhost kernel: [ 6656.906943]     0     9      2 0x00000000
Jul 21 17:44:57 localhost kernel: [ 6656.906957] [<c080e9c7>] (__schedule) from [<c080ef2b>] (schedule+0x2f/0x68)
Jul 21 17:44:57 localhost kernel: [ 6656.906970] [<c080ef2b>] (schedule) from [<c08114e1>] (schedule_timeout+0x69/0x2e4)
Jul 21 17:44:57 localhost kernel: [ 6656.906982] [<c080e9c7>] (__schedule) from [<c080ef2b>] (schedule+0x2f/0x68)
Jul 21 17:44:57 localhost kernel: [ 6656.906994] [<c080ef2b>] (schedule) from [<c08114e1>] (schedule_timeout+0x69/0x2e4)
Jul 21 17:44:57 localhost kernel: [ 6656.907004] [<c08114e1>] (schedule_timeout) from [<c0165869>] (rcu_gp_kthread+0x405/0x638)
Jul 21 17:44:57 localhost kernel: [ 6656.907016] [<c0165869>] (rcu_gp_kthread) from [<c0130569>] (kthread+0xfd/0x104)
Jul 21 17:44:57 localhost kernel: [ 6656.907026] [<c08114e1>] (schedule_timeout) from [<c0165869>] (rcu_gp_kthread+0x405/0x638)
Jul 21 17:44:57 localhost kernel: [ 6656.907037] [<c0165869>] (rcu_gp_kthread) from [<c0130569>] (kthread+0xfd/0x104)
Jul 21 17:44:57 localhost kernel: [ 6656.907045] [<c0130569>] (kthread) from [<c01010f9>] (ret_from_fork+0x11/0x38)
Jul 21 17:44:57 localhost kernel: [ 6656.907049] Exception stack(0xef117fb0 to 0xef117ff8)
Jul 21 17:44:57 localhost kernel: [ 6656.907057] [<c0130569>] (kthread) from [<c01010f9>] (ret_from_fork+0x11/0x38)
Jul 21 17:44:57 localhost kernel: [ 6656.907062] Exception stack(0xef117fb0 to 0xef117ff8)
Jul 21 17:44:57 localhost kernel: [ 6656.907064] 7fa0:                                     00000000 00000000 00000000 00000000
Jul 21 17:44:57 localhost kernel: [ 6656.907072] 7fa0:                                     00000000 00000000 00000000 00000000
Jul 21 17:44:57 localhost kernel: [ 6656.907075] 7fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul 21 17:44:57 localhost kernel: [ 6656.907082] 7fe0: 00000000 00000000 00000000 00000000 00000013 00000000
Jul 21 17:44:57 localhost kernel: [ 6656.907085] 7fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul 21 17:44:57 localhost kernel: [ 6656.907090] NMI backtrace for cpu 1
Jul 21 17:44:57 localhost kernel: [ 6656.907093] 7fe0: 00000000 00000000 00000000 00000000 00000013 00000000
Jul 21 17:44:57 localhost kernel: [ 6656.907100] CPU: 1 PID: 479 Comm: rs:main Q:Reg Not tainted 4.17.7-sunxi #125
Jul 21 17:44:57 localhost kernel: [ 6656.907103] Hardware name: Allwinner sun8i Family
Jul 21 17:44:57 localhost kernel: [ 6656.907115] [<c010cef9>] (unwind_backtrace) from [<c010a471>] (show_stack+0x11/0x14)
Jul 21 17:44:57 localhost kernel: [ 6656.907127] [<c010a471>] (show_stack) from [<c08006b1>] (dump_stack+0x69/0x78)
Jul 21 17:44:57 localhost kernel: [ 6656.907138] [<c08006b1>] (dump_stack) from [<c08043ff>] (nmi_cpu_backtrace+0xd3/0xd4)
Jul 21 17:44:57 localhost kernel: [ 6656.907150] [<c08043ff>] (nmi_cpu_backtrace) from [<c080448d>] (nmi_trigger_cpumask_backtrace+0x8d/0xc4)
Jul 21 17:44:57 localhost kernel: [ 6656.907163] [<c080448d>] (nmi_trigger_cpumask_backtrace) from [<c016766d>] (rcu_dump_cpu_stacks+0x77/0x94)
Jul 21 17:44:57 localhost kernel: [ 6656.907177] [<c016766d>] (rcu_dump_cpu_stacks) from [<c0166df1>] (rcu_check_callbacks+0x461/0x630)
Jul 21 17:44:57 localhost kernel: [ 6656.907190] [<c0166df1>] (rcu_check_callbacks) from [<c016b063>] (update_process_times+0x2b/0x48)
Jul 21 17:44:57 localhost kernel: [ 6656.907205] [<c016b063>] (update_process_times) from [<c01784eb>] (tick_sched_timer+0x37/0x74)
Jul 21 17:44:57 localhost kernel: [ 6656.907218] [<c01784eb>] (tick_sched_timer) from [<c016b89b>] (__hrtimer_run_queues+0xff/0x214)
Jul 21 17:44:57 localhost kernel: [ 6656.907230] [<c016b89b>] (__hrtimer_run_queues) from [<c016c321>] (hrtimer_interrupt+0xb5/0x1fc)
Jul 21 17:44:57 localhost kernel: [ 6656.907245] [<c016c321>] (hrtimer_interrupt) from [<c06cc119>] (arch_timer_handler_phys+0x25/0x28)
Jul 21 17:44:57 localhost kernel: [ 6656.907260] [<c06cc119>] (arch_timer_handler_phys) from [<c015ea3f>] (handle_percpu_devid_irq+0x5f/0x19c)
Jul 21 17:44:57 localhost kernel: [ 6656.907272] [<c015ea3f>] (handle_percpu_devid_irq) from [<c015b13d>] (generic_handle_irq+0x1d/0x28)
Jul 21 17:44:57 localhost kernel: [ 6656.907284] [<c015b13d>] (generic_handle_irq) from [<c015b5dd>] (__handle_domain_irq+0x45/0x84)
Jul 21 17:44:57 localhost kernel: [ 6656.907297] [<c015b5dd>] (__handle_domain_irq) from [<c04dacb7>] (gic_handle_irq+0x43/0x74)
Jul 21 17:44:57 localhost kernel: [ 6656.907309] [<c04dacb7>] (gic_handle_irq) from [<c0101a25>] (__irq_svc+0x65/0x94)
Jul 21 17:44:57 localhost kernel: [ 6656.907313] Exception stack(0xed0e1d78 to 0xed0e1dc0)
Jul 21 17:44:57 localhost kernel: [ 6656.907319] 1d60:                                                       ffeee948 b621d5d4
Jul 21 17:44:57 localhost kernel: [ 6656.907330] 1d80: 00000618 3132206c 3a373120 313a3434 6f6c2037 686c6163 0000736f 656b2074
Jul 21 17:44:57 localhost kernel: [ 6656.907341] 1da0: ed0e1f00 ed0e1f08 6c656e72 ed0e1dc8 205b203a c07fd916 200f0033 ffffffff
Jul 21 17:44:57 localhost kernel: [ 6656.907352] [<c0101a25>] (__irq_svc) from [<c07fd916>] (arm_copy_from_user+0x2f6/0x420)
Jul 21 17:44:57 localhost kernel: [ 6656.907366] [<c07fd916>] (arm_copy_from_user) from [<c049a5bd>] (copyin+0x35/0x44)
Jul 21 17:44:57 localhost kernel: [ 6656.907378] [<c049a5bd>] (copyin) from [<c049af65>] (iov_iter_copy_from_user_atomic+0x95/0x228)
Jul 21 17:44:57 localhost kernel: [ 6656.907393] [<c049af65>] (iov_iter_copy_from_user_atomic) from [<c01cf1eb>] (generic_perform_write+0xa3/0x13c)
Jul 21 17:44:57 localhost kernel: [ 6656.907408] [<c01cf1eb>] (generic_perform_write) from [<c01d0cc9>] (__generic_file_write_iter+0xc1/0x14c)
Jul 21 17:44:57 localhost kernel: [ 6656.907423] [<c01d0cc9>] (__generic_file_write_iter) from [<c01d0e63>] (generic_file_write_iter+0x10f/0x1ac)
Jul 21 17:44:57 localhost kernel: [ 6656.907439] [<c01d0e63>] (generic_file_write_iter) from [<c021d9e5>] (new_sync_write+0x7d/0xa0)
Jul 21 17:44:57 localhost kernel: [ 6656.907453] [<c021d9e5>] (new_sync_write) from [<c021f547>] (vfs_write+0x77/0x144)
Jul 21 17:44:57 localhost kernel: [ 6656.907463] [<c021f547>] (vfs_write) from [<c021f6e5>] (ksys_write+0x31/0x6c)
Jul 21 17:44:57 localhost kernel: [ 6656.907473] [<c021f6e5>] (ksys_write) from [<c0101001>] (ret_fast_syscall+0x1/0x62)
Jul 21 17:44:57 localhost kernel: [ 6656.907478] Exception stack(0xed0e1fa8 to 0xed0e1ff0)
Jul 21 17:44:57 localhost kernel: [ 6656.907487] 1fa0:                   0000091c b621d5b0 00000006 b621d5b0 0000091c 0047a970
Jul 21 17:44:57 localhost kernel: [ 6656.907497] 1fc0: 0000091c b621d5b0 00000000 00000004 fffff829 0047efe8 b60fe924 00492348
Jul 21 17:44:57 localhost kernel: [ 6656.907504] 1fe0: 00000000 b60fe4d8 00000000 b6ea33d0
Jul 21 17:44:57 localhost kernel: [ 6656.907514] Sending NMI from CPU 1 to CPUs 2:
Jul 21 17:44:57 localhost kernel: [ 6666.908355] Sending NMI from CPU 1 to CPUs 3:
Jul 21 17:44:57 localhost kernel: [ 6676.909196] Sending NMI from CPU 2 to CPUs 1:
Jul 21 17:44:57 localhost kernel: [ 6686.910036] NMI backtrace for cpu 2
Jul 21 17:44:57 localhost kernel: [ 6686.910043] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.17.7-sunxi #125
Jul 21 17:44:57 localhost kernel: [ 6686.910047] Hardware name: Allwinner sun8i Family
Jul 21 17:44:57 localhost kernel: [ 6686.910060] [<c010cef9>] (unwind_backtrace) from [<c010a471>] (show_stack+0x11/0x14)
Jul 21 17:44:57 localhost kernel: [ 6686.910071] [<c010a471>] (show_stack) from [<c08006b1>] (dump_stack+0x69/0x78)
Jul 21 17:44:57 localhost kernel: [ 6686.910083] [<c08006b1>] (dump_stack) from [<c08043ff>] (nmi_cpu_backtrace+0xd3/0xd4)
Jul 21 17:44:57 localhost kernel: [ 6686.910094] [<c08043ff>] (nmi_cpu_backtrace) from [<c080448d>] (nmi_trigger_cpumask_backtrace+0x8d/0xc4)
Jul 21 17:44:57 localhost kernel: [ 6686.910106] [<c080448d>] (nmi_trigger_cpumask_backtrace) from [<c016766d>] (rcu_dump_cpu_stacks+0x77/0x94)
Jul 21 17:44:57 localhost kernel: [ 6686.910121] [<c016766d>] (rcu_dump_cpu_stacks) from [<c0166df1>] (rcu_check_callbacks+0x461/0x630)
Jul 21 17:44:57 localhost kernel: [ 6686.910134] [<c0166df1>] (rcu_check_callbacks) from [<c016b063>] (update_process_times+0x2b/0x48)
Jul 21 17:44:57 localhost kernel: [ 6686.910147] [<c016b063>] (update_process_times) from [<c01784eb>] (tick_sched_timer+0x37/0x74)
Jul 21 17:44:57 localhost kernel: [ 6686.910160] [<c01784eb>] (tick_sched_timer) from [<c016b89b>] (__hrtimer_run_queues+0xff/0x214)
Jul 21 17:44:57 localhost kernel: [ 6686.910172] [<c016b89b>] (__hrtimer_run_queues) from [<c016c321>] (hrtimer_interrupt+0xb5/0x1fc)
Jul 21 17:44:57 localhost kernel: [ 6686.910186] [<c016c321>] (hrtimer_interrupt) from [<c06cc119>] (arch_timer_handler_phys+0x25/0x28)
Jul 21 17:44:57 localhost kernel: [ 6686.910200] [<c06cc119>] (arch_timer_handler_phys) from [<c015ea3f>] (handle_percpu_devid_irq+0x5f/0x19c)
Jul 21 17:44:57 localhost kernel: [ 6686.910212] [<c015ea3f>] (handle_percpu_devid_irq) from [<c015b13d>] (generic_handle_irq+0x1d/0x28)
Jul 21 17:44:57 localhost kernel: [ 6686.910224] [<c015b13d>] (generic_handle_irq) from [<c015b5dd>] (__handle_domain_irq+0x45/0x84)
Jul 21 17:44:57 localhost kernel: [ 6686.910236] [<c015b5dd>] (__handle_domain_irq) from [<c04dacb7>] (gic_handle_irq+0x43/0x74)
Jul 21 17:44:57 localhost kernel: [ 6686.910248] [<c04dacb7>] (gic_handle_irq) from [<c0101a25>] (__irq_svc+0x65/0x94)
Jul 21 17:44:57 localhost kernel: [ 6686.910253] Exception stack(0xef11ff60 to 0xef11ffa8)
Jul 21 17:44:57 localhost kernel: [ 6686.910263] ff60: 00000000 0042db1c ef6c3428 c0115981 ffffe000 c0d04d28 c0d04d78 00000004
Jul 21 17:44:57 localhost kernel: [ 6686.910273] ff80: 00000000 c0cad540 00000000 c0dd199f 00000000 ef11ffb0 c0107773 c0107774
Jul 21 17:44:57 localhost kernel: [ 6686.910278] ffa0: 40070033 ffffffff
Jul 21 17:44:57 localhost kernel: [ 6686.910291] [<c0101a25>] (__irq_svc) from [<c0107774>] (arch_cpu_idle+0x28/0x2c)
Jul 21 17:44:57 localhost kernel: [ 6686.910303] [<c0107774>] (arch_cpu_idle) from [<c013c001>] (do_idle+0x131/0x1b0)
Jul 21 17:44:57 localhost kernel: [ 6686.910313] [<c013c001>] (do_idle) from [<c013c255>] (cpu_startup_entry+0x19/0x1c)
Jul 21 17:44:57 localhost kernel: [ 6686.910322] [<c013c255>] (cpu_startup_entry) from [<40102451>] (0x40102451)
Jul 21 17:44:57 localhost kernel: [ 6686.910330] Sending NMI from CPU 2 to CPUs 3:
Jul 21 17:44:57 localhost kernel: [ 6696.911175] NMI backtrace for cpu 3
Jul 21 17:44:57 localhost kernel: [ 6696.911184] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.17.7-sunxi #125
Jul 21 17:44:57 localhost kernel: [ 6696.911186] Hardware name: Allwinner sun8i Family
Jul 21 17:44:57 localhost kernel: [ 6696.911195] PC is at _raw_spin_unlock_irqrestore+0x1c/0x20
Jul 21 17:44:57 localhost kernel: [ 6696.911202] LR is at rcu_process_callbacks+0x309/0x3d0
Jul 21 17:44:57 localhost kernel: [ 6696.911206] pc : [<c0812058>]    lr : [<c01661ad>]    psr: 000f0133
Jul 21 17:44:57 localhost kernel: [ 6696.911210] sp : ef121e78  ip : 0000000e  fp : c0d11440
Jul 21 17:44:57 localhost kernel: [ 6696.911213] r10: 00000008  r9 : c0d11840  r8 : 00000000
Jul 21 17:44:57 localhost kernel: [ 6696.911218] r7 : 600f0113  r6 : c0d11440  r5 : c0d11440  r4 : ef6d9a00
Jul 21 17:44:57 localhost kernel: [ 6696.911222] r3 : 00006489  r2 : 600f0113  r1 : 600f0113  r0 : c0d11440
Jul 21 17:44:57 localhost kernel: [ 6696.911227] Flags: nzcv  IRQs on  FIQs on  Mode SVC_32  ISA Thumb  Segment none
Jul 21 17:44:57 localhost kernel: [ 6696.911232] Control: 50c5387d  Table: 6d0e406a  DAC: 00000051
Jul 21 17:44:57 localhost kernel: [ 6696.911238] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.17.7-sunxi #125
Jul 21 17:44:57 localhost kernel: [ 6696.911241] Hardware name: Allwinner sun8i Family
Jul 21 17:44:57 localhost kernel: [ 6696.911254] [<c010cef9>] (unwind_backtrace) from [<c010a471>] (show_stack+0x11/0x14)
Jul 21 17:44:57 localhost kernel: [ 6696.911265] [<c010a471>] (show_stack) from [<c08006b1>] (dump_stack+0x69/0x78)
Jul 21 17:44:57 localhost kernel: [ 6696.911276] [<c08006b1>] (dump_stack) from [<c08043b7>] (nmi_cpu_backtrace+0x8b/0xd4)
Jul 21 17:44:57 localhost kernel: [ 6696.911287] [<c08043b7>] (nmi_cpu_backtrace) from [<c010c127>] (handle_IPI+0x9f/0x244)
Jul 21 17:44:57 localhost kernel: [ 6696.911299] [<c010c127>] (handle_IPI) from [<c04dace7>] (gic_handle_irq+0x73/0x74)
Jul 21 17:44:57 localhost kernel: [ 6696.911310] [<c04dace7>] (gic_handle_irq) from [<c0101a25>] (__irq_svc+0x65/0x94)
Jul 21 17:44:57 localhost kernel: [ 6696.911314] Exception stack(0xef121e28 to 0xef121e70)
Jul 21 17:44:57 localhost kernel: [ 6696.911322] 1e20:                   c0d11440 600f0113 600f0113 00006489 ef6d9a00 c0d11440
Jul 21 17:44:57 localhost kernel: [ 6696.911331] 1e40: c0d11440 600f0113 00000000 c0d11840 00000008 c0d11440 0000000e ef121e78
Jul 21 17:44:57 localhost kernel: [ 6696.911337] 1e60: c01661ad c0812058 000f0133 ffffffff
Jul 21 17:44:57 localhost kernel: [ 6696.911348] [<c0101a25>] (__irq_svc) from [<c0812058>] (_raw_spin_unlock_irqrestore+0x1c/0x20)
Jul 21 17:44:57 localhost kernel: [ 6696.911361] [<c0812058>] (_raw_spin_unlock_irqrestore) from [<c01661ad>] (rcu_process_callbacks+0x309/0x3d0)
Jul 21 17:44:57 localhost kernel: [ 6696.911372] [<c01661ad>] (rcu_process_callbacks) from [<c010221d>] (__do_softirq+0xc5/0x254)
Jul 21 17:44:57 localhost kernel: [ 6696.911386] [<c010221d>] (__do_softirq) from [<c011e95f>] (irq_exit+0x77/0x98)
Jul 21 17:44:57 localhost kernel: [ 6696.911398] [<c011e95f>] (irq_exit) from [<c015b5e1>] (__handle_domain_irq+0x49/0x84)
Jul 21 17:44:57 localhost kernel: [ 6696.911410] [<c015b5e1>] (__handle_domain_irq) from [<c04dacb7>] (gic_handle_irq+0x43/0x74)
Jul 21 17:44:57 localhost kernel: [ 6696.911421] [<c04dacb7>] (gic_handle_irq) from [<c0101a25>] (__irq_svc+0x65/0x94)
Jul 21 17:44:57 localhost kernel: [ 6696.911424] Exception stack(0xef121f60 to 0xef121fa8)
Jul 21 17:44:57 localhost kernel: [ 6696.911434] 1f60: 00000000 00430708 ef6d4428 c0115981 ffffe000 c0d04d28 c0d04d78 00000008
Jul 21 17:44:57 localhost kernel: [ 6696.911443] 1f80: 00000000 c0cad540 00000000 c0dd199f 00000000 ef121fb0 c0107773 c0107774
Jul 21 17:44:57 localhost kernel: [ 6696.911448] 1fa0: 400f0033 ffffffff
Jul 21 17:44:57 localhost kernel: [ 6696.911465] [<c0101a25>] (__irq_svc) from [<c0107774>] (arch_cpu_idle+0x28/0x2c)
Jul 21 17:44:57 localhost kernel: [ 6696.911478] [<c0107774>] (arch_cpu_idle) from [<c013c001>] (do_idle+0x131/0x1b0)
Jul 21 17:44:57 localhost kernel: [ 6696.911489] [<c013c001>] (do_idle) from [<c013c255>] (cpu_startup_entry+0x19/0x1c)
Jul 21 17:44:57 localhost kernel: [ 6696.911500] [<c013c255>] (cpu_startup_entry) from [<40102451>] (0x40102451)
Jul 21 17:45:01 localhost CRON[3045]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Jul 21 17:46:16 localhost systemd[1]: systemd-udevd.service: Watchdog timeout (limit 3min)!
Jul 21 17:46:16 localhost systemd[1]: systemd-udevd.service: Killing process 264 (systemd-udevd) with signal SIGABRT.
Jul 21 17:47:47 localhost systemd[1]: systemd-udevd.service: State 'stop-sigabrt' timed out. Terminating.
Jul 21 17:49:17 localhost systemd[1]: systemd-udevd.service: State 'stop-sigterm' timed out. Killing.
Jul 21 17:49:17 localhost systemd[1]: systemd-udevd.service: Killing process 264 (systemd-udevd) with signal SIGKILL.
Jul 21 17:50:47 localhost systemd[1]: systemd-udevd.service: Processes still around after SIGKILL. Ignoring.
Jul 21 17:52:17 localhost systemd[1]: systemd-udevd.service: State 'stop-final-sigterm' timed out. Killing.
Jul 21 17:52:17 localhost systemd[1]: systemd-udevd.service: Killing process 264 (systemd-udevd) with signal SIGKILL.
Jul 21 17:53:48 localhost systemd[1]: systemd-udevd.service: Processes still around after final SIGKILL. Entering failed mode.
Jul 21 17:53:48 localhost systemd[1]: systemd-udevd.service: Unit entered failed state.
Jul 21 17:53:48 localhost systemd[1]: systemd-udevd.service: Failed with result 'watchdog'.
Jul 21 17:53:48 localhost systemd[1]: systemd-udevd.service: Service has no hold-off time, scheduling restart.
Jul 21 17:53:48 localhost systemd[1]: Stopped udev Kernel Device Manager.
Jul 21 17:53:48 localhost systemd[1]: Starting udev Kernel Device Manager...
Jul 21 17:53:48 localhost systemd[1]: Started udev Kernel Device Manager.
Jul 21 17:55:01 localhost CRON[3069]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug  4 04:37:16 localhost kernel: [ 7341.823288] INFO: rcu_sched self-detected stall on CPU
Aug  4 04:37:16 localhost kernel: [ 7341.823314]     0-...!: (1 GPs behind) idle=656/0/1 softirq=298934/298935 fqs=0
Aug  4 04:37:16 localhost kernel: [ 7341.823317]      (t=106512892 jiffies g=255721 c=255720 q=3565)
Aug  4 04:37:16 localhost kernel: [ 7341.823335] rcu_sched kthread starved for 106512892 jiffies! g255721 c255720 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x402 ->cpu=1
Aug  4 04:37:16 localhost kernel: [ 7341.823338] RCU grace-period kthread stack dump:
Aug  4 04:37:16 localhost kernel: [ 7341.823342] rcu_sched       I    0     9      2 0x00000000
Aug  4 04:37:16 localhost kernel: [ 7341.823382] [<c080e9c7>] (__schedule) from [<c080ef2b>] (schedule+0x2f/0x68)
Aug  4 04:37:16 localhost kernel: [ 7341.823397] [<c080ef2b>] (schedule) from [<c08114e1>] (schedule_timeout+0x69/0x2e4)
Aug  4 04:37:16 localhost kernel: [ 7341.823414] [<c08114e1>] (schedule_timeout) from [<c0165869>] (rcu_gp_kthread+0x405/0x638)
Aug  4 04:37:16 localhost kernel: [ 7341.823430] [<c0165869>] (rcu_gp_kthread) from [<c0130569>] (kthread+0xfd/0x104)
Aug  4 04:37:16 localhost kernel: [ 7341.823443] [<c0130569>] (kthread) from [<c01010f9>] (ret_from_fork+0x11/0x38)
Aug  4 04:37:16 localhost kernel: [ 7341.823449] Exception stack(0xef117fb0 to 0xef117ff8)
Aug  4 04:37:16 localhost kernel: [ 7341.823457] 7fa0:                                     00000000 00000000 00000000 00000000
Aug  4 04:37:16 localhost kernel: [ 7341.823466] 7fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug  4 04:37:16 localhost kernel: [ 7341.823474] 7fe0: 00000000 00000000 00000000 00000000 00000013 00000000
Aug  4 04:37:16 localhost kernel: [ 7341.823493] NMI backtrace for cpu 0
Aug  4 04:37:16 localhost kernel: [ 7341.823502] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.17.7-sunxi #125
Aug  4 04:37:16 localhost kernel: [ 7341.823506] Hardware name: Allwinner sun8i Family
Aug  4 04:37:16 localhost kernel: [ 7341.823522] [<c010cef9>] (unwind_backtrace) from [<c010a471>] (show_stack+0x11/0x14)
Aug  4 04:37:16 localhost kernel: [ 7341.823536] [<c010a471>] (show_stack) from [<c08006b1>] (dump_stack+0x69/0x78)
Aug  4 04:37:16 localhost kernel: [ 7341.823549] [<c08006b1>] (dump_stack) from [<c08043ff>] (nmi_cpu_backtrace+0xd3/0xd4)
Aug  4 04:37:16 localhost kernel: [ 7341.823561] [<c08043ff>] (nmi_cpu_backtrace) from [<c080448d>] (nmi_trigger_cpumask_backtrace+0x8d/0xc4)
Aug  4 04:37:16 localhost kernel: [ 7341.823575] [<c080448d>] (nmi_trigger_cpumask_backtrace) from [<c016766d>] (rcu_dump_cpu_stacks+0x77/0x94)
Aug  4 04:37:16 localhost kernel: [ 7341.823591] [<c016766d>] (rcu_dump_cpu_stacks) from [<c0166df1>] (rcu_check_callbacks+0x461/0x630)
Aug  4 04:37:16 localhost kernel: [ 7341.823606] [<c0166df1>] (rcu_check_callbacks) from [<c016b063>] (update_process_times+0x2b/0x48)
Aug  4 04:37:16 localhost kernel: [ 7341.823622] [<c016b063>] (update_process_times) from [<c01784eb>] (tick_sched_timer+0x37/0x74)
Aug  4 04:37:16 localhost kernel: [ 7341.823637] [<c01784eb>] (tick_sched_timer) from [<c016b89b>] (__hrtimer_run_queues+0xff/0x214)
Aug  4 04:37:16 localhost kernel: [ 7341.823650] [<c016b89b>] (__hrtimer_run_queues) from [<c016c321>] (hrtimer_interrupt+0xb5/0x1fc)
Aug  4 04:37:16 localhost kernel: [ 7341.823667] [<c016c321>] (hrtimer_interrupt) from [<c06cc119>] (arch_timer_handler_phys+0x25/0x28)
Aug  4 04:37:16 localhost kernel: [ 7341.823685] [<c06cc119>] (arch_timer_handler_phys) from [<c015ea3f>] (handle_percpu_devid_irq+0x5f/0x19c)
Aug  4 04:37:16 localhost kernel: [ 7341.823698] [<c015ea3f>] (handle_percpu_devid_irq) from [<c015b13d>] (generic_handle_irq+0x1d/0x28)
Aug  4 04:37:16 localhost kernel: [ 7341.823711] [<c015b13d>] (generic_handle_irq) from [<c015b5dd>] (__handle_domain_irq+0x45/0x84)
Aug  4 04:37:16 localhost kernel: [ 7341.823726] [<c015b5dd>] (__handle_domain_irq) from [<c04dacb7>] (gic_handle_irq+0x43/0x74)
Aug  4 04:37:16 localhost kernel: [ 7341.823740] [<c04dacb7>] (gic_handle_irq) from [<c0101a25>] (__irq_svc+0x65/0x94)
Aug  4 04:37:16 localhost kernel: [ 7341.823745] Exception stack(0xc0d01f28 to 0xc0d01f70)
Aug  4 04:37:16 localhost kernel: [ 7341.823754] 1f20:                   00000000 01421650 ef6a1428 c0115981 ffffe000 c0d04d28
Aug  4 04:37:16 localhost kernel: [ 7341.823764] 1f40: c0d04d78 00000001 00000000 c0cad540 00000000 c0dd199f ffffffff c0d01f78
Aug  4 04:37:16 localhost kernel: [ 7341.823771] 1f60: c0107773 c0107774 400f0033 ffffffff
Aug  4 04:37:16 localhost kernel: [ 7341.823785] [<c0101a25>] (__irq_svc) from [<c0107774>] (arch_cpu_idle+0x28/0x2c)
Aug  4 04:37:16 localhost kernel: [ 7341.823800] [<c0107774>] (arch_cpu_idle) from [<c013c001>] (do_idle+0x131/0x1b0)
Aug  4 04:37:16 localhost kernel: [ 7341.823811] [<c013c001>] (do_idle) from [<c013c255>] (cpu_startup_entry+0x19/0x1c)
Aug  4 04:37:16 localhost kernel: [ 7341.823824] [<c013c255>] (cpu_startup_entry) from [<c0c00b5f>] (start_kernel+0x35d/0x36e)
 

Spoiler

Aug  4 05:05:35 localhost kernel: [ 9041.966131] sun6i-rtc 1f00000.rtc: rtc only supports year in range 1970 - 2033
Aug  4 05:05:43 localhost kernel: [ 9049.971115] sun6i-rtc 1f00000.rtc: rtc only supports year in range 1970 - 2033
Aug  4 05:05:51 localhost kernel: [ 9057.974120] sun6i-rtc 1f00000.rtc: rtc only supports year in range 1970 - 2033

 

 the last log entitys are spammed with rtc errors.

 

have changed the /etc/default/cpufrequtils to:

 

Spoiler

# WARNING: this file will be replaced on board support package (linux-root-...) upgrade
ENABLE=true
MIN_SPEED=408000
MAX_SPEED=500000
#MAX_SPEED=1296000
GOVERNOR=ondemand

since lower speed may helps.

 

[René Kliment]

Yeah, so can confirm the thing with my log too.

 

[314330.237556] INFO: rcu_sched detected stalls on CPUs/tasks:
[314330.243170]         (detected by 2, t=107378770 jiffies, g=3815634, c=3815633, q=30640)
[314330.250673] All QSes seen, last rcu_sched kthread activity 107378777 (421779489-314400712), jiffies_till_next_fqs=3, root ->qsmask 0x0
[314330.263135] rcu_sched kthread starved for 107378790 jiffies! g3815634 c3815633 f0x2 RCU_GP_WAIT_FQS(3) ->state=0x0 ->cpu=2
[314330.274266] RCU grace-period kthread stack dump:
[314330.237556] INFO: rcu_sched detected stalls on CPUs/tasks:
[314330.243170]         (detected by 2, t=107378770 jiffies, g=3815634, c=3815633, q=30640)
[314330.250673] All QSes seen, last rcu_sched kthread activity 107378777 (421779489-314400712), jiffies_till_next_fqs=3, root ->qsmask 0x0
[314330.262839] mysqld          R  running task        0  2081   1613 0x00000002
[314330.262877] [<c010cef9>] (unwind_backtrace) from [<c010a471>] (show_stack+0x11/0x14)
[314330.262892] [<c010a471>] (show_stack) from [<c0166fbd>] (rcu_check_callbacks+0x62d/0x630)
[314330.262905] [<c0166fbd>] (rcu_check_callbacks) from [<c016b063>] (update_process_times+0x2b/0x48)
[314330.262918] [<c016b063>] (update_process_times) from [<c01784eb>] (tick_sched_timer+0x37/0x74)
[314330.262930] [<c01784eb>] (tick_sched_timer) from [<c016b89b>] (__hrtimer_run_queues+0xff/0x214)
[314330.262939] [<c016b89b>] (__hrtimer_run_queues) from [<c016c321>] (hrtimer_interrupt+0xb5/0x1fc)
[314330.262954] [<c016c321>] (hrtimer_interrupt) from [<c06cc119>] (arch_timer_handler_phys+0x25/0x28)
[314330.262968] [<c06cc119>] (arch_timer_handler_phys) from [<c015ea3f>] (handle_percpu_devid_irq+0x5f/0x19c)
[314330.262978] [<c015ea3f>] (handle_percpu_devid_irq) from [<c015b13d>] (generic_handle_irq+0x1d/0x28)
[314330.262988] [<c015b13d>] (generic_handle_irq) from [<c015b5dd>] (__handle_domain_irq+0x45/0x84)
[314330.263001] [<c015b5dd>] (__handle_domain_irq) from [<c04dacb7>] (gic_handle_irq+0x43/0x74)
[314330.263012] [<c04dacb7>] (gic_handle_irq) from [<c0101a25>] (__irq_svc+0x65/0x94)
[314330.263017] Exception stack(0xe7347e40 to 0xe7347e88)
[314330.263025] 7e40: 00000000 00000000 ffff0001 e7c9b280 e7c9b210 e7347f30 e7347f18 00000000
[314330.263032] 7e60: e9a3e840 ee273000 e7c9b280 e7c9b080 e7c9b210 e7347e90 c03c0a87 c0154b28
[314330.263037] 7e80: 60080033 ffffffff
[314330.263050] [<c0101a25>] (__irq_svc) from [<c0154b28>] (down_write_trylock+0x28/0x48)
[314330.263063] [<c0154b28>] (down_write_trylock) from [<c03c0a87>] (btrfs_file_write_iter+0x5f/0x4d0)
[314330.263078] [<c03c0a87>] (btrfs_file_write_iter) from [<c021d9e5>] (new_sync_write+0x7d/0xa0)
[314330.263088] [<c021d9e5>] (new_sync_write) from [<c021f547>] (vfs_write+0x77/0x144)
[314330.263096] [<c021f547>] (vfs_write) from [<c021f7df>] (ksys_pwrite64+0x4b/0x5c)
[314330.263104] [<c021f7df>] (ksys_pwrite64) from [<c0101001>] (ret_fast_syscall+0x1/0x62)
[314330.263107] Exception stack(0xe7347fa8 to 0xe7347ff0)
[314330.263113] 7fa0:                   00100000 00000000 00000003 a5004000 00004000 00000000
[314330.263121] 7fc0: 00100000 00000000 00004000 000000b5 00000003 00000000 00a8990c a5004000
[314330.263126] 7fe0: 000000b5 a47fe8f8 b6ed419b b6ed5456
[314330.263135] rcu_sched kthread starved for 107378790 jiffies! g3815634 c3815633 f0x2 RCU_GP_WAIT_FQS(3) ->state=0x0 ->cpu=2
[314330.274266] RCU grace-period kthread stack dump:
[314330.278970] rcu_sched       R  running task        0     9      2 0x00000000
[314330.278994] [<c080e9c7>] (__schedule) from [<c080ef2b>] (schedule+0x2f/0x68)
[314330.279004] [<c080ef2b>] (schedule) from [<c08114e1>] (schedule_timeout+0x69/0x2e4)
[314330.279017] [<c08114e1>] (schedule_timeout) from [<c0165869>] (rcu_gp_kthread+0x405/0x638)
[314330.279028] [<c0165869>] (rcu_gp_kthread) from [<c0130569>] (kthread+0xfd/0x104)
[314330.279036] [<c0130569>] (kthread) from [<c01010f9>] (ret_from_fork+0x11/0x38)
[314330.279040] Exception stack(0xee917fb0 to 0xee917ff8)
[314330.279045] 7fa0:                                     00000000 00000000 00000000 00000000
[314330.279051] 7fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[314330.279057] 7fe0: 00000000 00000000 00000000 00000000 00000013 00000000

 

[AndrewT]

Same problem here:

 

Banana PI M1 - ARMBIAN 5.38 stable Debian GNU/Linux 9 (stretch) 4.14.18-sunxi

 

It makes a daily reboot and runs for a couple of days, maybe 3 to 5.

Then the system partial stops to work.

SSH login works, but my cronjobs don't run.

And it seems that the system is just using one of the two cpu cores.

 

When I try to restart the system with "reboot", it doesn't come up again.

I have to use the reset-button.

 

Here is my log:

Aug 12 21:51:26 pi-router kernel: [82260.470855] INFO: rcu_sched detected stalls on CPUs/tasks:
Aug 12 21:51:26 pi-router kernel: [82260.470892]        1-...: (1 GPs behind) idle=fe6/140000000000000/0 softirq=172179/172185 fqs=639049
Aug 12 21:51:26 pi-router kernel: [82260.470897]        (detected by 0, t=1578352 jiffies, g=83018, c=83017, q=198500)
Aug 12 21:51:26 pi-router kernel: [82260.470922] Sending NMI from CPU 0 to CPUs 1:
Aug 12 21:51:27 pi-router kernel: [82271.651723] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:51:29 pi-router kernel: [82273.811868] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:51:31 pi-router kernel: [82275.972016] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:51:34 pi-router kernel: [82278.132167] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:51:36 pi-router kernel: [82280.292319] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:51:38 pi-router kernel: [82282.452471] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:51:40 pi-router kernel: [82284.612622] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:51:42 pi-router kernel: [82286.772783] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:51:44 pi-router kernel: [82288.932919] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:51:46 pi-router kernel: [82291.093111] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:51:49 pi-router kernel: [82293.253221] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:51:51 pi-router kernel: [82295.413375] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:51:53 pi-router kernel: [82297.573525] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:51:55 pi-router kernel: [82299.733682] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:51:57 pi-router kernel: [82301.893902] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:51:59 pi-router kernel: [82304.053975] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:02 pi-router kernel: [82306.214126] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:04 pi-router kernel: [82308.374273] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:06 pi-router kernel: [82310.534438] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:08 pi-router kernel: [82312.694576] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:10 pi-router kernel: [82314.854736] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:12 pi-router kernel: [82317.015819] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:15 pi-router kernel: [82319.175042] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:17 pi-router kernel: [82321.335187] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:19 pi-router kernel: [82323.495331] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:29 pi-router kernel: [82323.525252] INFO: rcu_sched detected stalls on CPUs/tasks:
Aug 12 21:52:29 pi-router kernel: [82323.525290]        1-...: (1 GPs behind) idle=fe6/140000000000000/0 softirq=172179/172185 fqs=641602
Aug 12 21:52:29 pi-router kernel: [82323.525295]        (detected by 0, t=1584657 jiffies, g=83018, c=83017, q=199148)
Aug 12 21:52:29 pi-router kernel: [82323.525319] Sending NMI from CPU 0 to CPUs 1:
Aug 12 21:52:30 pi-router kernel: [82334.696137] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:32 pi-router kernel: [82336.856262] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:34 pi-router kernel: [82339.016434] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:37 pi-router kernel: [82341.176561] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:39 pi-router kernel: [82343.336713] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:41 pi-router kernel: [82345.496864] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:43 pi-router kernel: [82347.657017] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:45 pi-router kernel: [82349.817192] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:47 pi-router kernel: [82351.977312] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:50 pi-router kernel: [82354.137477] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:52 pi-router kernel: [82356.297613] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:54 pi-router kernel: [82358.457766] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:56 pi-router kernel: [82360.617913] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:58 pi-router kernel: [82362.778074] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:53:00 pi-router kernel: [82364.938289] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:53:02 pi-router kernel: [82367.098371] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:53:05 pi-router kernel: [82369.258521] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:53:07 pi-router kernel: [82371.418683] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:53:09 pi-router kernel: [82373.578825] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:53:11 pi-router kernel: [82375.738982] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:53:13 pi-router kernel: [82377.899121] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:53:15 pi-router kernel: [82380.060167] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:53:18 pi-router kernel: [82382.219429] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:53:20 pi-router kernel: [82384.379579] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:53:22 pi-router kernel: [82386.539739] thermal thermal_zone0: failed to read out thermal zone (-110)

 

[René Kliment]

I've been running 4.17.14-sunxi for 4 days straight now and it's crunching two `openssl speed` in a loop at 1200 MHz. So far so good.

Maybe this patch has something to do with it? https://github.com/armbian/build/commit/3326ccc11648e5ff482102ec401b22cc795006ae

I don't have time to compare with and without, but I'm super glad it works now :-)

[menno]

On 8/24/2018 at 2:09 PM, René Kliment said:

I've been running 4.17.14-sunxi for 4 days straight now and it's crunching two `openssl speed` in a loop at 1200 MHz. So far so good.

Maybe this patch has something to do with it? https://github.com/armbian/build/commit/3326ccc11648e5ff482102ec401b22cc795006ae

I don't have time to compare with and without, but I'm super glad it works now :-)

I was having very similar issues with my Orange Pi+ 2E and even with all patches applied (I'm running megous' mainline kernel[1] that includes the patch you linked) and I was still having trouble with random hangs and rcu stalls. I was able to get the machine stable by setting cpufreq to a fixed frequency. It does not matter if I set it to low or high so mine's now stable running at 1.37Ghz for days.

 

Did you lock yours at 1200Mhz or was it just never idle ? (Is it still stable for you?)

 

[1] https://github.com/megous/linux/

[piknew]

On 9/30/2018 at 7:20 PM, menno said:

I was able to get the machine stable by setting cpufreq to a fixed frequency. It does not matter if I set it to low or high so mine's now stable running at 1.37Ghz for days.

 

Could you please point out how to achieve this? All my H3 (also H2+) devices suffer from the issue as described. Even with newest kernel. So, after reading I decided to test stability with "constant cpu frequency".

 

EDIT: I've done it by:

 

[root@PKTEST ~]# cat /etc/default/cpufrequtils
# WARNING: this file will be replaced on board support package (linux-root-...) upgrade
ENABLE=true
MIN_SPEED=408000
MAX_SPEED=1296000
#GOVERNOR=ondemand
GOVERNOR=performance
[root@PKTEST ~]#

 

Question: would it be possible not to overwrite cpufrequtils file once support package is installed/upgraded?

[piknew]

Could you please take a look here:

 

Considering that when running on performance mode - system seems to be stable but once switched to "ondemand" - unstable. May the "mismatch" be the root cause here?

 

[Igor]

4 minutes ago, piknew said:

May the "mismatch" be the root cause here?

Yes it can be. It was noticed on pure mainline builds as well.

 

On 10/10/2018 at 7:14 PM, piknew said:

Question: would it be possible not to overwrite cpufrequtils file once support package is installed/upgraded?

This part is due to rework but since developers count is low, it takes a lot of time to get there.

[piknew]

For now I have modified cpufrequtils as following for my three boards (orangepipc, orangepiplus 2 GB, orangepiplus2e):

[root@PKBACKUP ~]# cat /etc/default/cpufrequtils
# WARNING: this file will be replaced on board support package (linux-root-...) upgrade
ENABLE=true
MIN_SPEED=480000
MAX_SPEED=1296000
GOVERNOR=ondemand
#GOVERNOR=performance

 

For orangepizero as following:

[root@PKOTHER ~]# cat /etc/default/cpufrequtils
# WARNING: this file will be replaced on board support package (linux-root-...) upgrade
ENABLE=true
MIN_SPEED=240000
MAX_SPEED=1200000
GOVERNOR=ondemand
#GOVERNOR=performance

 

Only orangepiplus2e is somehow loaded. With my own backup software, which is run twice a day - but in the past it was enough to freeze the board after a few days.

 

I will update if the change helped or not.

 

Question: I have noticed that orangepizero is using lower "MAX" settings. I understand as this device had an issue with overheating. How about other platforms, why lowest frequency settings is 480 MHz, not 240 MHz (which is also supported by H3)?

 

Some results for my platforms (please note that 816 MHz is common lowest choice for H3):

 

orangepiplu2e:

[root@PKBACKUP ~]# armbianmonitor -m
Stop monitoring using [ctrl]-[c]
Time        CPU    load %cpu %sys %usr %nice %io %irq   CPU  C.St.

16:36:53: 1296MHz  0.00   6%   1%   2%   0%   1%   0% 42.5°C  0/9
16:36:58:  816MHz  0.00   0%   0%   0%   0%   0%   0% 43.0°C  0/9
16:37:03:  816MHz  0.00   0%   0%   0%   0%   0%   0% 42.8°C  0/9
16:37:08:  816MHz  0.00   0%   0%   0%   0%   0%   0% 42.8°C  0/9
16:37:14:  816MHz  0.00   0%   0%   0%   0%   0%   0% 40.2°C  0/9
16:37:19:  816MHz  0.00   0%   0%   0%   0%   0%   0% 43.4°C  0/9
16:37:24:  816MHz  0.00   0%   0%   0%   0%   0%   0% 41.7°C  0/9
16:37:29:  816MHz  0.00   0%   0%   0%   0%   0%   0% 41.6°C  0/9
16:37:34:  816MHz  0.00   0%   0%   0%   0%   0%   0% 42.5°C  0/9^C

 

orangepiplus:

[root@PKHELPER ~]# armbianmonitor -m
Stop monitoring using [ctrl]-[c]
Time        CPU    load %cpu %sys %usr %nice %io %irq   CPU  C.St.

16:39:34:  816MHz  0.13   0%   0%   0%   0%   0%   0% 44.2°C  0/9
16:39:39: 1296MHz  0.18   0%   0%   0%   0%   0%   0% 43.8°C  0/9
16:39:44:  816MHz  0.17   0%   0%   0%   0%   0%   0% 43.9°C  0/9
16:39:49: 1296MHz  0.16   0%   0%   0%   0%   0%   0% 43.7°C  0/9
16:39:54:  816MHz  0.14   0%   0%   0%   0%   0%   0% 44.0°C  0/9
16:39:59: 1296MHz  0.13   0%   0%   0%   0%   0%   0% 44.3°C  0/9
16:40:04:  816MHz  0.12   0%   0%   0%   0%   0%   0% 44.3°C  0/9
16:40:09:  816MHz  0.11   0%   0%   0%   0%   0%   0% 43.7°C  0/9
16:40:15:  816MHz  0.10   0%   0%   0%   0%   0%   0% 43.1°C  0/9
16:40:20:  816MHz  0.09   0%   0%   0%   0%   0%   0% 43.7°C  0/9^C

 

orangepipc:

[root@PKTEST ~]# armbianmonitor -m
Stop monitoring using [ctrl]-[c]
Time        CPU    load %cpu %sys %usr %nice %io %irq   CPU  C.St.

16:41:11: 1296MHz  0.07   0%   0%   0%   0%   0%   0% 35.7°C  0/9
16:41:16:  816MHz  0.06   0%   0%   0%   0%   0%   0% 36.7°C  0/9
16:41:21:  816MHz  0.06   0%   0%   0%   0%   0%   0% 36.8°C  0/9
16:41:26:  816MHz  0.05   0%   0%   0%   0%   0%   0% 35.7°C  0/9
16:41:31:  816MHz  0.05   0%   0%   0%   0%   0%   0% 35.9°C  0/9
16:41:37:  816MHz  0.04   0%   0%   0%   0%   0%   0% 36.5°C  0/9
16:41:42:  816MHz  0.04   0%   0%   0%   0%   0%   0% 35.8°C  0/9
16:41:47:  816MHz  0.04   0%   0%   0%   0%   0%   0% 36.4°C  0/9
16:41:52:  816MHz  0.03   0%   0%   0%   0%   0%   0% 36.2°C  0/9^C

orangepizero:

[root@PKOTHER ~]# armbianmonitor -m
Stop monitoring using [ctrl]-[c]
Time        CPU    load %cpu %sys %usr %nice %io %irq   CPU  C.St.

16:42:33: 1200MHz  0.00   0%   0%   0%   0%   0%   0% 43.0°C  0/8
16:42:38:  648MHz  0.00   0%   0%   0%   0%   0%   0% 43.1°C  0/8
16:42:43:  648MHz  0.00   0%   0%   0%   0%   0%   0% 43.1°C  0/8
16:42:48:  648MHz  0.00   0%   0%   0%   0%   0%   0% 42.6°C  0/8
16:42:54:  648MHz  0.08   0%   0%   0%   0%   0%   0% 42.5°C  0/8
16:42:59:  648MHz  0.07   0%   0%   0%   0%   0%   0% 43.6°C  0/8
16:43:04:  648MHz  0.06   0%   0%   0%   0%   0%   0% 43.0°C  0/8
16:43:09:  648MHz  0.06   0%   0%   0%   0%   0%   0% 42.6°C  0/8^C

 

 

[rufik]

why lowest frequency settings is 480 MHz, not 240 MHz (which is also supported by H3)?

Because everything lower than 480 trashes SD card performance, as@tkaiser wrote a few times.

[piknew]

After correcting cpufrequtils file - no issues so far:

 

[root@PKBACKUP ~]# date
Fri Oct 26 11:03:47 CEST 2018
[root@PKBACKUP ~]# uptime
 11:03:53 up 4 days, 16:14,  1 user,  load average: 0.00, 0.00, 0.00
[root@PKBACKUP ~]# ll /etc/default/cpufrequtils
-r--r--r-- 1 root root 175 Oct 18 18:01 /etc/default/cpufrequtils
[root@PKBACKUP ~]# cat /etc/default/cpufrequtils
# WARNING: this file will be replaced on board support package (linux-root-...) upgrade
ENABLE=true
MIN_SPEED=480000
MAX_SPEED=1296000
GOVERNOR=ondemand
#GOVERNOR=performance
[root@PKBACKUP ~]#

Anybody else (who's SBC was impacted by this issue) can verify the same?

[piknew]

Still OK (I can confirm this for all of my "orange" SBCs):

 

[root@PKBACKUP ~]# date
Mon Oct 29 20:21:27 CET 2018
[root@PKBACKUP ~]# uptime
 20:21:32 up 8 days,  2:32,  1 user,  load average: 0.10, 0.03, 0.01
[root@PKBACKUP ~]# ll /etc/default/cpufrequtils
-r--r--r-- 1 root root 175 Oct 18 18:01 /etc/default/cpufrequtils
[root@PKBACKUP ~]# cat /etc/default/cpufrequtils
# WARNING: this file will be replaced on board support package (linux-root-...) upgrade
ENABLE=true
MIN_SPEED=480000
MAX_SPEED=1296000
GOVERNOR=ondemand
#GOVERNOR=performance
[root@PKBACKUP ~]#

 

[sunzone]

Unfortunately I gets the same "rcu_sched detected stalls on CPUs/tasks" error even with corrected "/etc/default/cpufrequtils" file

 

Spoiler

2020-08-06T13:30:01+09:00 10.8.2.182 CRON[14672]: pam_unix(cron:session): session opened for user root by (uid=0)
2020-08-06T13:30:01+09:00 10.8.2.182 CRON[14671]: pam_unix(cron:session): session opened for user root by (uid=0)
2020-08-06T13:30:01+09:00 10.8.2.182 CRON[14673]: (root) CMD (/root/logDasIpHealthInfo.sh)
2020-08-06T13:30:01+09:00 10.8.2.182 CRON[14674]: (root) CMD (/usr/lib/armbian/armbian-truncate-logs)
2020-08-06T13:30:01+09:00 10.8.2.182 CRON[14672]: pam_unix(cron:session): session closed for user root
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999178] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999216] rcu: #0110-...!: (2 ticks this GP) idle=77a/0/0x1 softirq=31006567/31006567 fqs=1
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999221] rcu: #011(detected by 2, t=44023 jiffies, g=104505893, q=78)
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999243] Sending NMI from CPU 2 to CPUs 0:
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999661] NMI backtrace for cpu 0
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999666] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G           O      4.19.62-sunxi #5.92
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999669] Hardware name: Allwinner sun8i Family
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999672] PC is at neigh_timer_handler+0x19e/0x1a4
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999675] LR is at 0x1cecf462
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999678] pc : [<c07f4886>]    lr : [<1cecf462>]    psr: 00010133
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999681] sp : c0e01de0  ip : d72bc300  fp : c0e04d70
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999684] r10: 1f19f000  r9 : 00000001  r8 : c9f84014
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999687] r7 : 00000002  r6 : c9f84018  r5 : c9f84000  r4 : c9f84030
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999691] r3 : c07f3025  r2 : 00000004  r1 : 000004e2  r0 : c0ab89b8
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999694] Flags: nzcv  IRQs on  FIQs on  Mode SVC_32  ISA Thumb  Segment none
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999697] Control: 50c5387d  Table: 411c406a  DAC: 00000051
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999701] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G           O      4.19.62-sunxi #5.92
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999704] Hardware name: Allwinner sun8i Family
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999708] [<c010d74d>] (unwind_backtrace) from [<c010a2f1>] (show_stack+0x11/0x14)
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999712] [<c010a2f1>] (show_stack) from [<c08fc121>] (dump_stack+0x69/0x78)
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999716] [<c08fc121>] (dump_stack) from [<c090036d>] (nmi_cpu_backtrace+0x59/0x90)
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999720] [<c090036d>] (nmi_cpu_backtrace) from [<c010c5d9>] (handle_IPI+0x85/0x2c0)
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999723] [<c010c5d9>] (handle_IPI) from [<c05c9c7f>] (gic_handle_irq+0x67/0x68)
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999727] [<c05c9c7f>] (gic_handle_irq) from [<c0101a65>] (__irq_svc+0x65/0x94)
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999730] Exception stack(0xc0e01d90 to 0xc0e01dd8)
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999734] 1d80:                                     c0ab89b8 000004e2 00000004 c07f3025
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999738] 1da0: c9f84030 c9f84000 c9f84018 00000002 c9f84014 00000001 1f19f000 c0e04d70
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999741] 1dc0: d72bc300 c0e01de0 1cecf462 c07f4886 00010133 ffffffff
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999745] [<c0101a65>] (__irq_svc) from [<c07f4886>] (neigh_timer_handler+0x19e/0x1a4)
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999749] [<c07f4886>] (neigh_timer_handler) from [<c0170037>] (call_timer_fn+0x23/0x118)
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999753] [<c0170037>] (call_timer_fn) from [<c01701bf>] (expire_timers+0x93/0xdc)
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999757] [<c01701bf>] (expire_timers) from [<c01702a3>] (run_timer_softirq+0x9b/0x138)
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999761] [<c01702a3>] (run_timer_softirq) from [<c010226d>] (__do_softirq+0xd5/0x27c)
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999765] [<c010226d>] (__do_softirq) from [<c011fefb>] (irq_exit+0x8f/0xc0)
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999769] [<c011fefb>] (irq_exit) from [<c015f855>] (__handle_domain_irq+0x49/0x84)
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999773] [<c015f855>] (__handle_domain_irq) from [<c05c9c51>] (gic_handle_irq+0x39/0x68)
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999776] [<c05c9c51>] (gic_handle_irq) from [<c0101a65>] (__irq_svc+0x65/0x94)
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999780] Exception stack(0xc0e01f18 to 0xc0e01f60)
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999783] 1f00:                                                       00000000 23391778
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999787] 1f20: dff55438 c0116441 ffffe000 c0e04d70 c0e04db8 00000001 00000000 c0e04d48
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999791] 1f40: c0dba870 00000000 c0e03d00 c0e01f68 c01078f3 c01078f4 40010033 ffffffff
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999795] [<c0101a65>] (__irq_svc) from [<c01078f4>] (arch_cpu_idle+0x28/0x2c)
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999799] [<c01078f4>] (arch_cpu_idle) from [<c013e96b>] (do_idle+0x14b/0x1d8)
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999803] [<c013e96b>] (do_idle) from [<c013ebed>] (cpu_startup_entry+0x19/0x1c)
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941308.999807] [<c013ebed>] (cpu_startup_entry) from [<c0d00bad>] (start_kernel+0x3a7/0x3c6)
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941309.000263] rcu: rcu_sched kthread starved for 44021 jiffies! g104505893 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=3
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941309.000266] rcu: RCU grace-period kthread stack dump:
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941309.000273] rcu_sched       R  running task        0    10      2 0x00000000
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941309.000299] [<c090acdb>] (__schedule) from [<c090b247>] (schedule+0x2f/0x68)
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941309.000314] [<c090b247>] (schedule) from [<c090daf7>] (schedule_timeout+0x77/0x320)
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941309.000333] [<c090daf7>] (schedule_timeout) from [<c016af6f>] (rcu_gp_kthread+0x41f/0x728)
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941309.000350] [<c016af6f>] (rcu_gp_kthread) from [<c0132ae9>] (kthread+0xfd/0x104)
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941309.000363] [<c0132ae9>] (kthread) from [<c01010f9>] (ret_from_fork+0x11/0x38)
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941309.000368] Exception stack(0xd7127fb0 to 0xd7127ff8)
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941309.000376] 7fa0:                                     00000000 00000000 00000000 00000000
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941309.000386] 7fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
2020-08-06T13:30:02+09:00 10.8.2.182 kernel: [1941309.000395] 7fe0: 00000000 00000000 00000000 00000000 00000013 00000000
2020-08-06T13:31:05+09:00 10.8.2.182 systemd[1]: systemd-logind.service: Watchdog timeout (limit 3min)!
2020-08-06T13:31:05+09:00 10.8.2.182 systemd[1]: systemd-logind.service: Killing process 708 (systemd-logind) with signal SIGABRT.
2020-08-06T13:31:05+09:00 10.8.2.182 systemd[1]: systemd-logind.service: Main process exited, code=killed, status=6/ABRT
2020-08-06T13:31:05+09:00 10.8.2.182 systemd[1]: systemd-logind.service: Failed with result 'watchdog'.
2020-08-06T13:31:05+09:00 10.8.2.182 systemd[1]: systemd-logind.service: Service has no hold-off time, scheduling restart.
2020-08-06T13:31:05+09:00 10.8.2.182 systemd[1]: systemd-logind.service: Scheduled restart job, restart counter is at 1.
2020-08-06T13:31:05+09:00 10.8.2.182 systemd[1]: Stopped Login Service.
2020-08-06T13:31:05+09:00 10.8.2.182 systemd[1]: Starting Login Service...
2020-08-06T13:31:05+09:00 10.8.2.182 systemd-logind[14740]: New seat seat0.
2020-08-06T13:31:05+09:00 10.8.2.182 systemd[1]: Started Login Service.
2020-08-06T13:34:05+09:00 10.8.2.182 systemd[1]: systemd-timesyncd.service: Watchdog timeout (limit 3min)!
2020-08-06T13:34:05+09:00 10.8.2.182 systemd[1]: systemd-timesyncd.service: Killing process 329 (systemd-timesyn) with signal SIGABRT.
2020-08-06T13:34:05+09:00 10.8.2.182 systemd[1]: systemd-resolved.service: Watchdog timeout (limit 3min)!
2020-08-06T13:34:05+09:00 10.8.2.182 systemd[1]: systemd-resolved.service: Killing process 336 (systemd-resolve) with signal SIGABRT.
2020-08-06T13:34:05+09:00 10.8.2.182 systemd[1]: systemd-logind.service: Watchdog timeout (limit 3min)!
2020-08-06T13:34:05+09:00 10.8.2.182 systemd[1]: systemd-logind.service: Killing process 14740 (systemd-logind) with signal SIGABRT.
2020-08-06T13:34:05+09:00 10.8.2.182 systemd[1]: systemd-logind.service: Main process exited, code=killed, status=6/ABRT
2020-08-06T13:34:05+09:00 10.8.2.182 systemd[1]: systemd-logind.service: Failed with result 'watchdog'.
2020-08-06T13:34:05+09:00 10.8.2.182 systemd[1]: systemd-logind.service: Service has no hold-off time, scheduling restart.
2020-08-06T13:34:05+09:00 10.8.2.182 systemd[1]: systemd-logind.service: Scheduled restart job, restart counter is at 2.
2020-08-06T13:34:05+09:00 10.8.2.182 systemd[1]: Stopped Login Service.
2020-08-06T13:34:05+09:00 10.8.2.182 systemd[1]: Starting Login Service...
2020-08-06T13:34:06+09:00 10.8.2.182 systemd-logind[14771]: New seat seat0.
2020-08-06T13:34:06+09:00 10.8.2.182 systemd[1]: Started Login Service.

This kernel hang occurred right after 2 CRON jobs started at the same time, hence CPU was not in idle at the time.

/etc/default/cpufrequtils 

# WARNING: this file will be replaced on board support package (linux-root-...) upgrade
ENABLE=true
MIN_SPEED=480000
MAX_SPEED=816000
GOVERNOR=ondemand

 

You can see armbian -U output here: 

https://paste.c-net.org/b1965858-723d-3409-2e8e-264e9e05d156

 

Running: 4.19.62-sunxi on Orange Pi Zero.

 

I am going to check with GOVERNOR=performance

Can anybody give me a hint/clues what might have caused this error and how to solve it?

 

Thank you

Edited December 8, 2021 by TRS-80
put long output inside spoiler

[Werner]

Quote

4.19.62-sunxi #5.92

Your board is way outdated and will not receive support. Please upgrade to a more recent kernel.

[Marmelade]

Armbian 21.08.1 Focal

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.2 LTS"

 

5.10.60-sunxi

Gets frozen after 10 min - 10h, while I run a bash script calling rather simple program written in C.

This GPS C program just reads GPS at /dev/ttyS2 at 9600bd

I have made a lot of work with the script and GPS-program having no luck but learned a lot.

 

Processor speed 480-1102MHz.

I took a photo of the frozen screen and made Google find this forum.

 

Here is the part of my script, which gets data  from the GPS-program:

 

timeout 3 /home/bin/i2cLCD/gps >$TMPFILE &
pid=$!
echo "PID: $pid"
wait $pid
read uline < $TMPFILE