Jump to content

OrangePi+ 2E (H3) crashes on 4.14 with INFO: rcu_sched self-detected stall on CPU

René Kliment

By René Kliment
in Allwinner sunxi

 Share

Recommended Posts

René Kliment

René Kliment

Posted
Posted

Hello everyone.
This happens on my OrangePi after a couple of hours of running. I don't think this happened before installing the kernel 4.14. I haven't changed my power supply, which should be working well - custom DC/DC converter connected to GPIO pins.

Is anyone else experiencing this? Anything I can do?
  

[19155.760977] INFO: rcu_sched self-detected stall on CPU
[19155.760983] INFO: rcu_sched self-detected stall on CPU
[19155.760997] INFO: rcu_sched detected stalls on CPUs/tasks:
[19155.761010]  2-...: (1 ticks this GP) idle=18a/1/0 softirq=332295/332295 fqs=0
[19155.761017]  0-...: (0 ticks this GP) idle=dbe/1/0 softirq=475103/475103 fqs=0
[19155.761018]
[19155.761025]  1-...: (1 GPs behind) idle=f36/1/0 softirq=422883/422884 fqs=0
[19155.761034]  2-...: (1 ticks this GP) idle=18a/1/0 softirq=332295/332295 fqs=0
[19155.761040] rcu_sched kthread starved for 91911 jiffies! g126395 c126394 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x402 ->cpu=3
[19155.761041]
[19155.819568]  1-...: (1 GPs behind) idle=f36/1/0 softirq=422883/422884 fqs=0
[19155.826612]   (t=91911 jiffies g=126395 c=126394 q=3)
[19155.831672] rcu_sched kthread starved for 91911 jiffies! g126395 c126394 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x0 ->cpu=0
[19185.762182] rcu_sched kthread starved for 91911 jiffies! g126395 c126394 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x0 ->cpu=0
[19155.760977] INFO: rcu_sched self-detected stall on CPU
[19155.760983] INFO: rcu_sched self-detected stall on CPU
[19155.760997] INFO: rcu_sched detected stalls on CPUs/tasks:
[19155.761010]  2-...: (1 ticks this GP) idle=18a/1/0 softirq=332295/332295 fqs=0 
[19155.761017]  0-...: (0 ticks this GP) idle=dbe/1/0 softirq=475103/475103 fqs=0 
[19155.761018] 
[19155.761025]  1-...: (1 GPs behind) idle=f36/1/0 softirq=422883/422884 fqs=0 
[19155.761028]  (t=91911 jiffies g=126395 c=126394 q=3)
[19155.761034]  2-...: (1 ticks this GP) idle=18a/1/0 softirq=332295/332295 fqs=0 
[19155.761040] rcu_sched kthread starved for 91911 jiffies! g126395 c126394 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x402 ->cpu=3
[19155.761041] 
[19155.761044] rcu_sched       I
[19155.761048] (detected by 3, t=91911 jiffies, g=126395, c=126394, q=3)
[19155.761052]     0     8      2 0x00000000
[19155.761057] Sending NMI from CPU 3 to CPUs 0:
[19155.761079] [<c08762b9>] (__schedule) from [<c087682b>] (schedule+0x2f/0x68)
[19155.761093] [<c087682b>] (schedule) from [<c087909d>] (schedule_timeout+0x75/0x314)
[19155.761108] [<c087909d>] (schedule_timeout) from [<c0164f95>] (rcu_gp_kthread+0x415/0x674)
[19155.761123] [<c0164f95>] (rcu_gp_kthread) from [<c0131a9d>] (kthread+0xfd/0x104)
[19155.761138] [<c0131a9d>] (kthread) from [<c0106719>] (ret_from_fork+0x11/0x38)
[19155.819568]  1-...: (1 GPs behind) idle=f36/1/0 softirq=422883/422884 fqs=0 
[19155.826612]   (t=91911 jiffies g=126395 c=126394 q=3)
[19155.831672] rcu_sched kthread starved for 91911 jiffies! g126395 c126394 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x0 ->cpu=0
[19155.842191] rcu_sched       R  running task        0     8      2 0x00000000
[19155.842210] [<c08762b9>] (__schedule) from [<c087682b>] (schedule+0x2f/0x68)
[19155.842222] [<c087682b>] (schedule) from [<c087909d>] (schedule_timeout+0x75/0x314)
[19155.842234] [<c087909d>] (schedule_timeout) from [<c0164f95>] (rcu_gp_kthread+0x415/0x674)
[19155.842245] [<c0164f95>] (rcu_gp_kthread) from [<c0131a9d>] (kthread+0xfd/0x104)
[19155.842256] [<c0131a9d>] (kthread) from [<c0106719>] (ret_from_fork+0x11/0x38)
[19165.761432] Sending NMI from CPU 3 to CPUs 1:
[19175.761806] Sending NMI from CPU 3 to CPUs 2:
[19185.762182] rcu_sched kthread starved for 91911 jiffies! g126395 c126394 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x0 ->cpu=0
[19185.762186] Sending NMI from CPU 2 to CPUs 0:
[19185.772707] rcu_sched       R  running task        0     8      2 0x00000000
[19185.772726] [<c08762b9>] (__schedule) from [<c087682b>] (schedule+0x2f/0x68)
[19185.772738] [<c087682b>] (schedule) from [<c087909d>] (schedule_timeout+0x75/0x314)
[19185.772749] [<c087909d>] (schedule_timeout) from [<c0164f95>] (rcu_gp_kthread+0x415/0x674)
[19185.772760] [<c0164f95>] (rcu_gp_kthread) from [<c0131a9d>] (kthread+0xfd/0x104)
[19185.772772] [<c0131a9d>] (kthread) from [<c0106719>] (ret_from_fork+0x11/0x38)
[19195.762559] Sending NMI from CPU 2 to CPUs 1:
[19205.762933] NMI backtrace for cpu 2
[19205.762942] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.14.14-sunxi #38
[19205.762945] Hardware name: Allwinner sun8i Family
[19205.762959] [<c010db15>] (unwind_backtrace) from [<c010a0d9>] (show_stack+0x11/0x14)
[19205.762970] [<c010a0d9>] (show_stack) from [<c0867e29>] (dump_stack+0x69/0x78)
[19205.762983] [<c0867e29>] (dump_stack) from [<c086bc37>] (nmi_cpu_backtrace+0xd3/0xd4)
[19205.762996] [<c086bc37>] (nmi_cpu_backtrace) from [<c086bccf>] (nmi_trigger_cpumask_backtrace+0x97/0xd0)
[19205.763008] [<c086bccf>] (nmi_trigger_cpumask_backtrace) from [<c01662a7>] (rcu_dump_cpu_stacks+0x77/0x94)
[19205.763019] [<c01662a7>] (rcu_dump_cpu_stacks) from [<c0165a65>] (rcu_check_callbacks+0x4d5/0x690)
[19205.763032] [<c0165a65>] (rcu_check_callbacks) from [<c0169f5f>] (update_process_times+0x2b/0x48)
[19205.763046] [<c0169f5f>] (update_process_times) from [<c0177d11>] (tick_sched_timer+0x31/0x68)
[19205.763057] [<c0177d11>] (tick_sched_timer) from [<c016ac85>] (__hrtimer_run_queues+0xf5/0x224)
[19205.763066] [<c016ac85>] (__hrtimer_run_queues) from [<c016af81>] (hrtimer_interrupt+0x81/0x180)
[19205.763079] [<c016af81>] (hrtimer_interrupt) from [<c07483a1>] (arch_timer_handler_phys+0x25/0x28)
[19205.763093] [<c07483a1>] (arch_timer_handler_phys) from [<c015d52f>] (handle_percpu_devid_irq+0x57/0x19c)
[19205.763107] [<c015d52f>] (handle_percpu_devid_irq) from [<c0159a49>] (generic_handle_irq+0x1d/0x28)
[19205.763120] [<c0159a49>] (generic_handle_irq) from [<c0159e59>] (__handle_domain_irq+0x45/0x84)
[19205.763131] [<c0159e59>] (__handle_domain_irq) from [<c01013b5>] (gic_handle_irq+0x39/0x68)
[19205.763141] [<c01013b5>] (gic_handle_irq) from [<c010aa25>] (__irq_svc+0x65/0x94)
[19205.763146] Exception stack(0xee523f78 to 0xee523fc0)
[19205.763152] 3f60:                                                       00000001 00000000
[19205.763161] 3f80: 00000000 c0116561 ffffe000 c0d03fcc c0d03f6c c0cb6438 c0ddd8eb 00000000
[19205.763170] 3fa0: 00000000 00000000 00087b5d ee523fc8 c01070e7 c01070e8 40000033 ffffffff
[19205.763181] [<c010aa25>] (__irq_svc) from [<c01070e8>] (arch_cpu_idle+0x28/0x2c)
[19205.763194] [<c01070e8>] (arch_cpu_idle) from [<c014c93d>] (do_idle+0x115/0x16c)
[19205.763206] [<c014c93d>] (do_idle) from [<c014cb89>] (cpu_startup_entry+0x19/0x1c)
[19205.763216] [<c014cb89>] (cpu_startup_entry) from [<401016f1>] (0x401016f1)
[19205.763226] Sending NMI from CPU 1 to CPUs 0:
[19215.763599] NMI backtrace for cpu 1
[19215.763605] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.14.14-sunxi #38
[19215.763608] Hardware name: Allwinner sun8i Family
[19215.763618] [<c010db15>] (unwind_backtrace) from [<c010a0d9>] (show_stack+0x11/0x14)
[19215.763627] [<c010a0d9>] (show_stack) from [<c0867e29>] (dump_stack+0x69/0x78)
[19215.763638] [<c0867e29>] (dump_stack) from [<c086bc37>] (nmi_cpu_backtrace+0xd3/0xd4)
[19215.763650] [<c086bc37>] (nmi_cpu_backtrace) from [<c086bccf>] (nmi_trigger_cpumask_backtrace+0x97/0xd0)
[19215.763661] [<c086bccf>] (nmi_trigger_cpumask_backtrace) from [<c01662a7>] (rcu_dump_cpu_stacks+0x77/0x94)
[19215.763672] [<c01662a7>] (rcu_dump_cpu_stacks) from [<c0165a65>] (rcu_check_callbacks+0x4d5/0x690)
[19215.763683] [<c0165a65>] (rcu_check_callbacks) from [<c0169f5f>] (update_process_times+0x2b/0x48)
[19215.763695] [<c0169f5f>] (update_process_times) from [<c0177d11>] (tick_sched_timer+0x31/0x68)
[19215.763705] [<c0177d11>] (tick_sched_timer) from [<c016ac85>] (__hrtimer_run_queues+0xf5/0x224)
[19215.763714] [<c016ac85>] (__hrtimer_run_queues) from [<c016af81>] (hrtimer_interrupt+0x81/0x180)
[19215.763724] [<c016af81>] (hrtimer_interrupt) from [<c07483a1>] (arch_timer_handler_phys+0x25/0x28)
[19215.763735] [<c07483a1>] (arch_timer_handler_phys) from [<c015d52f>] (handle_percpu_devid_irq+0x57/0x19c)
[19215.763746] [<c015d52f>] (handle_percpu_devid_irq) from [<c0159a49>] (generic_handle_irq+0x1d/0x28)
[19215.763758] [<c0159a49>] (generic_handle_irq) from [<c0159e59>] (__handle_domain_irq+0x45/0x84)
[19215.763768] [<c0159e59>] (__handle_domain_irq) from [<c01013b5>] (gic_handle_irq+0x39/0x68)
[19215.763777] [<c01013b5>] (gic_handle_irq) from [<c010aa25>] (__irq_svc+0x65/0x94)
[19215.763781] Exception stack(0xee521f78 to 0xee521fc0)
[19215.763787] 1f60:                                                       00000001 00000000
[19215.763796] 1f80: 00000000 c0116561 ffffe000 c0d03fcc c0d03f6c c0cb6438 c0ddd8eb 00000000
[19215.763805] 1fa0: 00000000 00000000 00000018 ee521fc8 c01070e7 c01070e8 40000033 ffffffff
[19215.763815] [<c010aa25>] (__irq_svc) from [<c01070e8>] (arch_cpu_idle+0x28/0x2c)
[19215.763826] [<c01070e8>] (arch_cpu_idle) from [<c014c93d>] (do_idle+0x115/0x16c)
[19215.763837] [<c014c93d>] (do_idle) from [<c014cb89>] (cpu_startup_entry+0x19/0x1c)
[19215.763846] [<c014cb89>] (cpu_startup_entry) from [<401016f1>] (0x401016f1)
[19215.763853] Sending NMI from CPU 1 to CPUs 2:
[19225.764386] NMI backtrace for cpu 2
[19225.764393] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.14.14-sunxi #38
[19225.764395] Hardware name: Allwinner sun8i Family
[19225.764399] task: ee4f8000 task.stack: ee522000
[19225.764404] PC is at __do_softirq+0x7a/0x25c
[19225.764412] LR is at irq_exit+0x7f/0xc4
[19225.764416] pc : [<c0101462>]    lr : [<c011ef33>]    psr: 40000133
[19225.764419] sp : ee523ee8  ip : 7fffffff  fp : 4000001f
[19225.764422] r10: c0d02080  r9 : ee434000  r8 : 00000001
[19225.764426] r7 : ffffe000  r6 : 00000282  r5 : 00000000  r4 : ffffe000
[19225.764430] r3 : 00000080  r2 : 00000000  r1 : c0df6740  r0 : c0df6740
[19225.764435] Flags: nZcv  IRQs on  FIQs on  Mode SVC_32  ISA Thumb  Segment none
[19225.764439] Control: 50c5387d  Table: 56ff006a  DAC: 00000051
[19225.764444] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.14.14-sunxi #38
[19225.764446] Hardware name: Allwinner sun8i Family
[19225.764456] [<c010db15>] (unwind_backtrace) from [<c010a0d9>] (show_stack+0x11/0x14)
[19225.764465] [<c010a0d9>] (show_stack) from [<c0867e29>] (dump_stack+0x69/0x78)
[19225.764475] [<c0867e29>] (dump_stack) from [<c086bbef>] (nmi_cpu_backtrace+0x8b/0xd4)
[19225.764486] [<c086bbef>] (nmi_cpu_backtrace) from [<c010ca01>] (handle_IPI+0x75/0x278)
[19225.764495] [<c010ca01>] (handle_IPI) from [<c01013e3>] (gic_handle_irq+0x67/0x68)
[19225.764502] [<c01013e3>] (gic_handle_irq) from [<c010aa25>] (__irq_svc+0x65/0x94)
[19225.764506] Exception stack(0xee523e98 to 0xee523ee0)
[19225.764510] 3e80:                                                       c0df6740 c0df6740
[19225.764519] 3ea0: 00000000 00000080 ffffe000 00000000 00000282 ffffe000 00000001 ee434000
[19225.764527] 3ec0: c0d02080 4000001f 7fffffff ee523ee8 c011ef33 c0101462 40000133 ffffffff
[19225.764535] [<c010aa25>] (__irq_svc) from [<c0101462>] (__do_softirq+0x7a/0x25c)
[19225.764545] [<c0101462>] (__do_softirq) from [<c011ef33>] (irq_exit+0x7f/0xc4)
[19225.764557] [<c011ef33>] (irq_exit) from [<c0159e5d>] (__handle_domain_irq+0x49/0x84)
[19225.764567] [<c0159e5d>] (__handle_domain_irq) from [<c01013b5>] (gic_handle_irq+0x39/0x68)
[19225.764575] [<c01013b5>] (gic_handle_irq) from [<c010aa25>] (__irq_svc+0x65/0x94)
[19225.764578] Exception stack(0xee523f78 to 0xee523fc0)
[19225.764582] 3f60:                                                       00000001 00000000
[19225.764590] 3f80: 00000000 c0116561 ffffe000 c0d03fcc c0d03f6c c0cb6438 c0ddd8eb 00000000
[19225.764603] 3fa0: 00000000 00000000 00087b5d ee523fc8 c01070e7 c01070e8 40000033 ffffffff
[19225.764613] [<c010aa25>] (__irq_svc) from [<c01070e8>] (arch_cpu_idle+0x28/0x2c)
[19225.764624] [<c01070e8>] (arch_cpu_idle) from [<c014c93d>] (do_idle+0x115/0x16c)
[19225.764642] [<c014c93d>] (do_idle) from [<c014cb89>] (cpu_startup_entry+0x19/0x1c)
[19225.764656] [<c014cb89>] (cpu_startup_entry) from [<401016f1>] (0x401016f1)

 

0
René Kliment

René Kliment

Posted
  • Author
Posted
  On 1/21/2018 at 3:58 PM, Igor said:


No, unknown to me. What kind of things were you doing on it? Or idle?

Expand  


Happens with both idle and busy Pi. I've been trying to rsync+ssh from a remote to an USB connected HDD. The HDD has its own power.
Also, the Pi has a heat sink, so no overheating either.

This is the weirdest thing ...

0
sermore

sermore

Posted
Posted

Hi everybody,

 

Same behavior here for a Lime2 running latest kernel: 

Welcome to ARMBIAN 5.38 stable Debian GNU/Linux 8 (jessie) 4.14.15-sunxi

After around 4 hours of idle, this errors start appearing crashing the board after a while.

  

[    0.430765] cpuidle: using governor ladder
[    0.430826] cpuidle: using governor menu
[    0.432049] hw-breakpoint: found 5 (+1 reserved) breakpoint and 4 watchpoint registers.
[    0.432085] hw-breakpoint: maximum watchpoint size is 8 bytes.
[    0.616165] raid6: int32x1  gen()   171 MB/s
[    0.786408] raid6: int32x1  xor()   145 MB/s
[    0.956533] raid6: int32x2  gen()   231 MB/s
[    1.126539] raid6: int32x2  xor()   175 MB/s
[    1.296737] raid6: int32x4  gen()   233 MB/s
[    1.466834] raid6: int32x4  xor()   174 MB/s
[    1.637134] raid6: int32x8  gen()   223 MB/s
[    1.807109] raid6: int32x8  xor()   155 MB/s
[    1.977297] raid6: neonx1   gen()   447 MB/s
[    2.147379] raid6: neonx1   xor()   417 MB/s
[    2.317515] raid6: neonx2   gen()   600 MB/s
[    2.487619] raid6: neonx2   xor()   537 MB/s
[    2.657756] raid6: neonx4   gen()   721 MB/s
[    2.827934] raid6: neonx4   xor()   606 MB/s
[    2.998142] raid6: neonx8   gen()   650 MB/s
[    3.168205] raid6: neonx8   xor()   548 MB/s
[    3.168222] raid6: using algorithm neonx4 gen() 721 MB/s
[    3.168236] raid6: .... xor() 606 MB/s, rmw enabled
[    3.168251] raid6: using neon recovery algorithm
[    3.168680] reg-fixed-voltage ahci-5v: could not find pctldev for node /soc@01c00000/pinctrl@01c20800/ahci_pwr_pin@1, deferring probe
[    3.168761] reg-fixed-voltage usb0-vbus: could not find pctldev for node /soc@01c00000/pinctrl@01c20800/usb0_vbus_pin@0, deferring probe
[    3.170286] SCSI subsystem initialized
[    3.170635] libata version 3.00 loaded.
[    3.170877] usbcore: registered new interface driver usbfs
[    3.170947] usbcore: registered new interface driver hub
[    3.171023] usbcore: registered new device driver usb
[    3.171233] media: Linux media interface: v0.10
[    3.171278] Linux video capture interface: v2.00
[    3.171354] pps_core: LinuxPPS API ver. 1 registered
[    3.171370] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    3.171406] PTP clock support registered
[    3.173002] clocksource: Switched to clocksource arch_sys_counter
[    3.276824] VFS: Disk quotas dquot_6.6.0
[    3.276954] VFS: Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
[    3.285576] NET: Registered protocol family 2
[    3.286305] TCP established hash table entries: 8192 (order: 3, 32768 bytes)
[    3.286426] TCP bind hash table entries: 8192 (order: 4, 65536 bytes)
[    3.286563] TCP: Hash tables configured (established 8192 bind 8192)
[    3.286699] UDP hash table entries: 512 (order: 2, 16384 bytes)
[    3.286774] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
[    3.287025] NET: Registered protocol family 1
[    3.287604] RPC: Registered named UNIX socket transport module.
[    3.287636] RPC: Registered udp transport module.
[    3.287650] RPC: Registered tcp transport module.
[    3.287664] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    3.287966] Trying to unpack rootfs image as initramfs...
[    3.602626] Freeing initrd memory: 4468K
[    3.603579] hw perfevents: no interrupt-affinity property for /pmu, guessing.
[    3.603963] hw perfevents: enabled with armv7_cortex_a7 PMU driver, 5 counters available
[    3.605010] audit: initializing netlink subsys (disabled)
[    3.605314] audit: type=2000 audit(3.590:1): state=initialized audit_enabled=0 res=1
[    3.605707] Initialise system trusted keyrings
[    3.605946] workingset: timestamp_bits=14 max_order=18 bucket_order=4
[    3.611182] zbud: loaded
[    3.613784] NFS: Registering the id_resolver key type
[    3.613844] Key type id_resolver registered
[    3.613860] Key type id_legacy registered
[    3.613886] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    3.613905] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[    3.615061] JFS: nTxBlock = 7996, nTxLock = 63973
[    3.625790] SGI XFS with ACLs, security attributes, realtime, no debug enabled
[    3.633462] Key type asymmetric registered
[    3.633577] bounce: pool size: 64 pages
[    3.633686] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247)
[    3.633890] io scheduler noop registered
[    3.633912] io scheduler deadline registered
[    3.634244] io scheduler cfq registered (default)
[    3.634264] io scheduler mq-deadline registered
[    3.634279] io scheduler kyber registered
[    3.634427] io scheduler bfq registered
[    3.635405] sun4i-usb-phy 1c13400.phy: could not find pctldev for node /soc@01c00000/pinctrl@01c20800/usb0_id_detect_pin@0, deferring probe
[    3.638749] sun4i-pinctrl 1c20800.pinctrl: initialized sunXi PIO driver
[    3.694001] Serial: 8250/16550 driver, 8 ports, IRQ sharing disabled
[    3.697445] console [ttyS0] disabled
[    3.717691] 1c28000.serial: ttyS0 at MMIO 0x1c28000 (irq = 43, base_baud = 1500000) is a U6_16550A
[    4.622634] console [ttyS0] enabled
[    4.630296] brd: module loaded
[    4.639839] loop: module loaded
[    4.645140] libphy: Fixed MDIO Bus: probed
[    4.649851] sun7i-dwmac 1c50000.ethernet: PTP uses main clock
[    4.655654] sun7i-dwmac 1c50000.ethernet: no reset control found
[    4.661705] sun7i-dwmac 1c50000.ethernet: no regulator found
[    4.667437] sun7i-dwmac 1c50000.ethernet: Ring mode enabled
[    4.673032] sun7i-dwmac 1c50000.ethernet: DMA HW capability register supported
[    4.680260] sun7i-dwmac 1c50000.ethernet: Normal descriptors
[    4.699525] libphy: stmmac: probed
[    4.702968] RTL8211B Gigabit Ethernet stmmac-0:01: attached PHY driver [RTL8211B Gigabit Ethernet] (mii_bus:phy_addr=stmmac-0:01, irq=POLL)
[    4.716645] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    4.723219] ehci-platform: EHCI generic platform driver
[    4.728865] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    4.735140] ohci-platform: OHCI generic platform driver
[    4.740799] usbcore: registered new interface driver usb-storage
[    4.747567] mousedev: PS/2 mouse device common for all mice
[    4.753996] sunxi-rtc 1c20d00.rtc: rtc core: registered rtc-sunxi as rtc0
[    4.760803] sunxi-rtc 1c20d00.rtc: RTC enabled
[    4.765428] i2c /dev entries driver
[    4.769889] axp20x-i2c 0-0034: AXP20x variant AXP209 found
[    4.783238] axp20x-gpio axp20x-gpio: AXP209 GPIO driver loaded
[    4.797380] input: axp20x-pek as /devices/platform/soc@01c00000/1c2ac00.i2c/i2c-0/0-0034/axp20x-pek/input/input0
[    4.808304] ldo1: supplied by regulator-dummy
[    4.812951] ldo2: supplied by regulator-dummy
[    4.818454] ldo3: supplied by regulator-dummy
[    4.823495] ldo4: supplied by regulator-dummy
[    4.828061] ldo5: supplied by regulator-dummy
[    4.833067] dcdc2: supplied by regulator-dummy
[    4.838195] dcdc3: supplied by regulator-dummy
[    4.846580] axp20x-i2c 0-0034: Backup (RTC) battery charging is disabled
[    4.853467] axp20x-i2c 0-0034: AXP20X driver loaded
[    4.861403] sunxi-wdt 1c20c90.watchdog: Watchdog enabled (timeout=16 sec, nowayout=0)
[    4.881744] sunxi-mmc 1c0f000.mmc: Got CD GPIO
[    4.944802] sunxi-mmc 1c0f000.mmc: base:0xf0f6a000 irq:27
[    4.963850] ledtrig-cpu: registered to indicate activity on CPUs
[    4.969993] hidraw: raw HID events driver (C) Jiri Kosina
[    4.975576] usbcore: registered new interface driver usbhid
[    4.981167] usbhid: USB HID core driver
[    4.993241] NET: Registered protocol family 10
[    5.028107] Segment Routing with IPv6
[    5.031942] NET: Registered protocol family 17
[    5.036712] Key type dns_resolver registered
[    5.041405] Registering SWP/SWPB emulation handler
[    5.047011] registered taskstats version 1
[    5.051146] Loading compiled-in X.509 certificates
[    5.056181] zswap: loaded using pool lzo/zbud
[    5.062555] Btrfs loaded, crc32c=crc32c-generic
[    5.071664] mmc0: host does not support reading read-only switch, assuming write-enable
[    5.076538] Key type encrypted registered
[    5.086486] mmc0: new high speed SDHC card at address 0002
[    5.092664] mmcblk0: mmc0:0002 00000 3.70 GiB 
[    5.098869]  mmcblk0: p1
[    5.183244] ahci-sunxi 1c18000.sata: controller can't do PMP, turning off CAP_PMP
[    5.190874] ahci-sunxi 1c18000.sata: SSS flag set, parallel bus scan disabled
[    5.198182] ahci-sunxi 1c18000.sata: AHCI 0001.0100 32 slots 1 ports 3 Gbps 0x1 impl platform mode
[    5.207249] ahci-sunxi 1c18000.sata: flags: ncq sntf stag pm led clo only pio slum part ccc 
[    5.233859] scsi host0: ahci-sunxi
[    5.237530] ata1: SATA max UDMA/133 mmio [mem 0x01c18000-0x01c18fff] port 0x100 irq 33
[    5.246127] ehci-platform 1c14000.usb: EHCI Host Controller
[    5.251749] ehci-platform 1c14000.usb: new USB bus registered, assigned bus number 1
[    5.259836] ehci-platform 1c14000.usb: irq 30, io mem 0x01c14000
[    5.293036] ehci-platform 1c14000.usb: USB 2.0 started, EHCI 1.00
[    5.299406] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[    5.306229] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    5.313467] usb usb1: Product: EHCI Host Controller
[    5.318348] usb usb1: Manufacturer: Linux 4.14.15-sunxi ehci_hcd
[    5.324367] usb usb1: SerialNumber: 1c14000.usb
[    5.334934] hub 1-0:1.0: USB hub found
[    5.338765] hub 1-0:1.0: 1 port detected
[    5.353270] ehci-platform 1c1c000.usb: EHCI Host Controller
[    5.358901] ehci-platform 1c1c000.usb: new USB bus registered, assigned bus number 2
[    5.367010] ehci-platform 1c1c000.usb: irq 34, io mem 0x01c1c000
[    5.403025] ehci-platform 1c1c000.usb: USB 2.0 started, EHCI 1.00
[    5.409398] usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
[    5.416221] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    5.423460] usb usb2: Product: EHCI Host Controller
[    5.428341] usb usb2: Manufacturer: Linux 4.14.15-sunxi ehci_hcd
[    5.434359] usb usb2: SerialNumber: 1c1c000.usb
[    5.444878] hub 2-0:1.0: USB hub found
[    5.448705] hub 2-0:1.0: 1 port detected
[    5.463202] ohci-platform 1c14400.usb: Generic Platform OHCI controller
[    5.469859] ohci-platform 1c14400.usb: new USB bus registered, assigned bus number 3
[    5.477918] ohci-platform 1c14400.usb: irq 31, io mem 0x01c14400
[    5.557589] usb usb3: New USB device found, idVendor=1d6b, idProduct=0001
[    5.564451] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    5.571680] usb usb3: Product: Generic Platform OHCI controller
[    5.575031] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[    5.583835] usb usb3: Manufacturer: Linux 4.14.15-sunxi ohci_hcd
[    5.589863] usb usb3: SerialNumber: 1c14400.usb
[    5.589878] ata1.00: ATA-10: ST4000LM024-2AN17V, 0001, max UDMA/133
[    5.589884] ata1.00: 7814037168 sectors, multi 0: LBA48 NCQ (depth 31/32)
[    5.608165] hub 3-0:1.0: USB hub found
[    5.611981] hub 3-0:1.0: 1 port detected
[    5.616872] ohci-platform 1c1c400.usb: Generic Platform OHCI controller
[    5.623598] ohci-platform 1c1c400.usb: new USB bus registered, assigned bus number 4
[    5.631624] ohci-platform 1c1c400.usb: irq 35, io mem 0x01c1c400
[    5.707745] usb usb4: New USB device found, idVendor=1d6b, idProduct=0001
[    5.714676] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    5.721954] usb usb4: Product: Generic Platform OHCI controller
[    5.727963] usb usb4: Manufacturer: Linux 4.14.15-sunxi ohci_hcd
[    5.734054] usb usb4: SerialNumber: 1c1c400.usb
[    5.746805] hub 4-0:1.0: USB hub found
[    5.750615] hub 4-0:1.0: 1 port detected
[    5.755727] usb_phy_generic usb_phy_generic.0.auto: usb_phy_generic.0.auto supply vcc not found, using dummy regulator
[    5.767076] musb-hdrc musb-hdrc.1.auto: MUSB HDRC host driver
[    5.772849] musb-hdrc musb-hdrc.1.auto: new USB bus registered, assigned bus number 5
[    5.781047] usb usb5: New USB device found, idVendor=1d6b, idProduct=0002
[    5.787903] usb usb5: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    5.795155] usb usb5: Product: MUSB HDRC host driver
[    5.800129] usb usb5: Manufacturer: Linux 4.14.15-sunxi musb-hcd
[    5.806170] usb usb5: SerialNumber: musb-hdrc.1.auto
[    5.811891] hub 5-0:1.0: USB hub found
[    5.815769] hub 5-0:1.0: 1 port detected
[    5.832503] of_cfs_init
[    5.835125] of_cfs_init: OK
[    5.838111] vcc3v0: disabling
[    5.840041] ata1.00: configured for UDMA/133
[    5.840770] scsi 0:0:0:0: Direct-Access     ATA      ST4000LM024-2AN1 0001 PQ: 0 ANSI: 5
[    5.841634] sd 0:0:0:0: Attached scsi generic sg0 type 0
[    5.842141] sd 0:0:0:0: [sda] 7814037168 512-byte logical blocks: (4.00 TB/3.64 TiB)
[    5.842148] sd 0:0:0:0: [sda] 4096-byte physical blocks
[    5.842210] sd 0:0:0:0: [sda] Write Protect is off
[    5.842217] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
[    5.842306] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    5.885690] vcc5v0: disabling
[    5.888678] vddio-csi1: disabling
[    5.892325] usb0-vbus: disabling
[    5.905456]  sda: sda1
[    5.911377] sd 0:0:0:0: [sda] Attached SCSI disk
[    5.924294] Freeing unused kernel memory: 1024K
[    6.004056] systemd-udevd[140]: starting version 215
[    6.120167] sunxi-mmc 1c11000.mmc: allocated mmc-pwrseq
[    6.123158] usb 3-1: new full-speed USB device number 2 using ohci-platform
[    6.183128] sunxi-mmc 1c11000.mmc: base:0xf101c000 irq:28
[    6.277175] mmc1: new DDR MMC card at address 0001
[    6.285756] mmcblk1: mmc1:0001 P1XXXX 3.60 GiB 
[    6.290895] mmcblk1boot0: mmc1:0001 P1XXXX partition 1 16.0 MiB
[    6.297649] mmcblk1boot1: mmc1:0001 P1XXXX partition 2 16.0 MiB
[    6.304808]  mmcblk1: p1
[    6.405184] usb 3-1: New USB device found, idVendor=0403, idProduct=6015
[    6.412110] usb 3-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[    6.419477] usb 3-1: Product: FT231X USB UART
[    6.424082] usb 3-1: Manufacturer: FTDI
[    6.428041] usb 3-1: SerialNumber: DN018W7V
[    9.348004] EXT4-fs (sda1): mounted filesystem with writeback data mode. Opts: (null)
[   10.398725] systemd[1]: systemd 215 running in system mode. (+PAM +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ -SECCOMP -APPARMOR)
[   10.412542] systemd[1]: Detected architecture 'arm'.
[   10.541809] systemd[1]: Set hostname to <lime2>.
[   12.644366] systemd[1]: Cannot add dependency job for unit display-manager.service, ignoring: Unit display-manager.service failed to load: No such file or directory.
[   12.663903] systemd[1]: Expecting device dev-ttyS0.device...
[   12.680433] systemd[1]: Starting Forward Password Requests to Wall Directory Watch.
[   12.688580] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
[   12.696363] systemd[1]: Starting Remote File Systems (Pre).
[   12.702351] systemd[1]: Reached target Remote File Systems (Pre).
[   12.708828] systemd[1]: Starting Arbitrary Executable File Formats File System Automount Point.
[   13.207363] fuse init (API version 7.26)
[   13.397347] Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
[   13.572762] systemd-udevd[232]: starting version 215
[   14.722137] sun4i-ss 1c15000.crypto-engine: Die ID 0
[   14.862608] usbcore: registered new interface driver usbserial
[   14.868722] usbcore: registered new interface driver usbserial_generic
[   14.875684] usbserial: USB Serial support registered for generic
[   14.940131] EXT4-fs (sda1): re-mounted. Opts: commit=600,errors=remount-ro
[   15.010305] usbcore: registered new interface driver ftdi_sio
[   15.016269] usbserial: USB Serial support registered for FTDI USB Serial Device
[   15.024084] ftdi_sio 3-1:1.0: FTDI USB Serial Device converter detected
[   15.031379] usb 3-1: Detected FT-X
[   15.059946] at24 1-0050: 2048 byte 24c16 EEPROM, writable, 16 bytes/write
[   15.169264] usb 3-1: FTDI USB Serial Device converter now attached to ttyUSB0
[   15.524996] random: crng init done
[   15.547002] Adding 131068k swap on /var/swap.  Priority:-2 extents:2 across:139260k FS
[   15.620089] EXT4-fs (mmcblk0p1): recovery complete
[   15.627993] EXT4-fs (mmcblk0p1): mounted filesystem with writeback data mode. Opts: (null)
[   16.669524] systemd-journald[244]: Received request to flush runtime journal from PID 1
[   17.123148] thermal thermal_zone0: failed to read out thermal zone (-110)
[   18.413470] RTL8211B Gigabit Ethernet stmmac-0:01: attached PHY driver [RTL8211B Gigabit Ethernet] (mii_bus:phy_addr=stmmac-0:01, irq=POLL)
[   18.427802] sun7i-dwmac 1c50000.ethernet eth0: RX IPC Checksum Offload disabled
[   18.435210] sun7i-dwmac 1c50000.ethernet eth0: No MAC Management Counters available
[   18.442879] sun7i-dwmac 1c50000.ethernet eth0: PTP not supported by HW
[   18.449784] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[   19.523959] sun7i-dwmac 1c50000.ethernet eth0: Link is Up - 1Gbps/Full - flow control rx/tx
[   19.532404] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[19229.640294] thermal thermal_zone0: failed to read out thermal zone (-110)
[19231.800287] thermal thermal_zone0: failed to read out thermal zone (-110)
[19233.960274] thermal thermal_zone0: failed to read out thermal zone (-110)
[19236.120261] thermal thermal_zone0: failed to read out thermal zone (-110)
[19238.280247] thermal thermal_zone0: failed to read out thermal zone (-110)
[19240.440232] thermal thermal_zone0: failed to read out thermal zone (-110)
[19242.600255] thermal thermal_zone0: failed to read out thermal zone (-110)
[19244.760203] thermal thermal_zone0: failed to read out thermal zone (-110)
[19246.920200] thermal thermal_zone0: failed to read out thermal zone (-110)
[19249.080193] thermal thermal_zone0: failed to read out thermal zone (-110)
[19249.650082] INFO: rcu_sched detected stalls on CPUs/tasks:
[19249.655608] 	0-...: (1 GPs behind) idle=116/140000000000000/0 softirq=211124/211128 fqs=1014 
[19249.664131] 	(detected by 1, t=2102 jiffies, g=119264, c=119263, q=101)
[19249.670755] Sending NMI from CPU 1 to CPUs 0:
[19259.900121] ata1.00: exception Emask 0x0 SAct 0x10000 SErr 0x0 action 0x6 frozen
[19259.907555] ata1.00: failed command: READ FPDMA QUEUED
[19259.912746] ata1.00: cmd 60/20:80:e0:02:0a/00:00:09:00:00/40 tag 16 ncq dma 16384 in
         res 40/00:00:00:4f:c2/00:00:00:00:00/40 Emask 0x4 (timeout)
[19259.927925] ata1.00: status: { DRDY }
[19259.931622] ata1: hard resetting link
[19260.261407] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[19260.840088] thermal thermal_zone0: failed to read out thermal zone (-110)
[19263.000097] thermal thermal_zone0: failed to read out thermal zone (-110)
[19265.160081] thermal thermal_zone0: failed to read out thermal zone (-110)
[19265.320044] ata1.00: qc timeout (cmd 0xec)
[19265.324172] ata1.00: failed to IDENTIFY (I/O error, err_mask=0x4)
[19265.330302] ata1.00: revalidation failed (errno=-5)
[19265.335195] ata1: hard resetting link
[19265.661344] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[19267.320065] thermal thermal_zone0: failed to read out thermal zone (-110)
[19269.480191] thermal thermal_zone0: failed to read out thermal zone (-110)
[19271.640031] thermal thermal_zone0: failed to read out thermal zone (-110)
[19273.800051] thermal thermal_zone0: failed to read out thermal zone (-110)
[19275.058307] systemd[1]: Starting Journal Service...
[19275.879982] ata1.00: qc timeout (cmd 0xec)
[19275.884118] ata1.00: failed to IDENTIFY (I/O error, err_mask=0x4)
[19275.890249] ata1.00: revalidation failed (errno=-5)
[19275.895136] ata1: limiting SATA link speed to 1.5 Gbps
[19275.900300] ata1: hard resetting link
[19275.960001] thermal thermal_zone0: failed to read out thermal zone (-110)
[19276.231277] ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
[19278.120001] thermal thermal_zone0: failed to read out thermal zone (-110)
[19280.279979] thermal thermal_zone0: failed to read out thermal zone (-110)
[19282.439970] thermal thermal_zone0: failed to read out thermal zone (-110)
[19284.599957] thermal thermal_zone0: failed to read out thermal zone (-110)
[19286.759938] thermal thermal_zone0: failed to read out thermal zone (-110)
[19288.919934] thermal thermal_zone0: failed to read out thermal zone (-110)
[19291.079917] thermal thermal_zone0: failed to read out thermal zone (-110)
[19293.239904] thermal thermal_zone0: failed to read out thermal zone (-110)
[19295.399901] thermal thermal_zone0: failed to read out thermal zone (-110)
[19297.559875] thermal thermal_zone0: failed to read out thermal zone (-110)
[19299.719863] thermal thermal_zone0: failed to read out thermal zone (-110)
[19301.879854] thermal thermal_zone0: failed to read out thermal zone (-110)
[19304.039836] thermal thermal_zone0: failed to read out thermal zone (-110)
[19306.199822] thermal thermal_zone0: failed to read out thermal zone (-110)
[19306.599785] ata1.00: qc timeout (cmd 0xec)
[19306.603916] ata1.00: failed to IDENTIFY (I/O error, err_mask=0x4)
[19306.610047] ata1.00: revalidation failed (errno=-5)
[19306.614929] ata1.00: disabled
[19306.617950] ata1: hard resetting link
[19306.951139] ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
[19306.957402] ata1: EH complete
[19306.960529] sd 0:0:0:0: [sda] tag#17 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[19306.968824] sd 0:0:0:0: [sda] tag#17 CDB: opcode=0x88 88 00 00 00 00 00 09 0a 02 e0 00 00 00 20 00 00
[19306.978068] print_req_error: I/O error, dev sda, sector 151651040
[19306.984514] sd 0:0:0:0: [sda] tag#18 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[19306.992833] sd 0:0:0:0: [sda] tag#18 CDB: opcode=0x88 88 00 00 00 00 00 09 0a 02 e8 00 00 00 08 00 00
[19307.002069] print_req_error: I/O error, dev sda, sector 151651048
[19307.009822] sd 0:0:0:0: [sda] tag#19 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[19307.018119] sd 0:0:0:0: [sda] tag#19 CDB: opcode=0x88 88 00 00 00 00 00 0d 9d 3f 58 00 00 00 08 00 00
[19307.027370] print_req_error: I/O error, dev sda, sector 228409176
[19307.033557] sd 0:0:0:0: [sda] tag#20 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[19307.041863] sd 0:0:0:0: [sda] tag#20 CDB: opcode=0x88 88 00 00 00 00 00 0d 9d 3f c0 00 00 00 08 00 00
[19307.051099] print_req_error: I/O error, dev sda, sector 228409280
[19307.057556] sd 0:0:0:0: [sda] tag#21 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[19307.065873] sd 0:0:0:0: [sda] tag#21 CDB: opcode=0x88 88 00 00 00 00 00 0d 9d 3f 58 00 00 00 08 00 00
[19307.075110] print_req_error: I/O error, dev sda, sector 228409176
[19307.086488] systemd[1]: systemd-journald.service: main process exited, code=killed, status=7/BUS
[19307.096874] systemd[1]: Unit systemd-journald.service entered failed state.
[19307.108866] systemd[1]: systemd-journald.service has no holdoff time, scheduling restart.
[19307.121960] systemd[1]: rsyslog.service: main process exited, code=killed, status=7/BUS
[19307.134009] systemd[1]: Unit rsyslog.service entered failed state.
[19307.159615] systemd[1]: Stopping Journal Service...
[19307.165033] systemd[1]: Starting Journal Service...
[19307.174569] systemd[1]: Started Journal Service.
[19307.185739] sd 0:0:0:0: [sda] tag#22 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[19307.194116] sd 0:0:0:0: [sda] tag#22 CDB: opcode=0x88 88 00 00 00 00 00 0d 9d 3f c0 00 00 00 08 00 00
[19307.203361] print_req_error: I/O error, dev sda, sector 228409280
[19307.345665] systemd[1]: systemd-journald.service has no holdoff time, scheduling restart.
[19307.354544] systemd[1]: rsyslog.service holdoff time over, scheduling restart.
[19307.383925] sd 0:0:0:0: [sda] tag#23 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[19307.392314] sd 0:0:0:0: [sda] tag#23 CDB: opcode=0x88 88 00 00 00 00 00 09 0a 44 10 00 00 00 20 00 00
[19307.401567] print_req_error: I/O error, dev sda, sector 151667728
[19307.419843] sd 0:0:0:0: [sda] tag#24 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[19307.428146] sd 0:0:0:0: [sda] tag#24 CDB: opcode=0x88 88 00 00 00 00 00 0d 9d 3f c0 00 00 00 08 00 00
[19307.437395] print_req_error: I/O error, dev sda, sector 228409280
[19307.448606] sd 0:0:0:0: [sda] tag#25 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[19307.456980] sd 0:0:0:0: [sda] tag#25 CDB: opcode=0x88 88 00 00 00 00 00 09 0a 44 10 00 00 00 08 00 00
[19307.466218] print_req_error: I/O error, dev sda, sector 151667728
[19307.543581] sd 0:0:0:0: [sda] tag#26 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
[19307.551956] sd 0:0:0:0: [sda] tag#26 CDB: opcode=0x88 88 00 00 00 00 00 0d 9d 3f c0 00 00 00 08 00 00
[19307.561199] print_req_error: I/O error, dev sda, sector 228409280
[19308.068643] systemd[7314]: Failed at step EXEC spawning /usr/sbin/rsyslogd: Input/output error
[19308.318928] systemd[7316]: Failed at step EXEC spawning /usr/sbin/rsyslogd: Input/output error
[19308.359849] thermal thermal_zone0: failed to read out thermal zone (-110)
[19308.568789] systemd[7318]: Failed at step EXEC spawning /usr/sbin/rsyslogd: Input/output error
[19310.519807] thermal thermal_zone0: failed to read out thermal zone (-110)
[19312.680555] thermal thermal_zone0: failed to read out thermal zone (-110)
[19312.699686] INFO: rcu_sched detected stalls on CPUs/tasks:
[19312.705188] 	0-...: (1 GPs behind) idle=116/140000000000000/0 softirq=211124/211128 fqs=3551 
[19312.713711] 	(detected by 1, t=8407 jiffies, g=119264, c=119263, q=3474)
[19312.720421] Sending NMI from CPU 1 to CPUs 0:
[19323.879702] thermal thermal_zone0: failed to read out thermal zone (-110)
[19326.039691] thermal thermal_zone0: failed to read out thermal zone (-110)
[19328.199684] thermal thermal_zone0: failed to read out thermal zone (-110)
[19330.359671] thermal thermal_zone0: failed to read out thermal zone (-110)
[19332.519654] thermal thermal_zone0: failed to read out thermal zone (-110)
[19334.679644] thermal thermal_zone0: failed to read out thermal zone (-110)
[19336.839654] thermal thermal_zone0: failed to read out thermal zone (-110)
[19338.999618] thermal thermal_zone0: failed to read out thermal zone (-110)
[19341.159594] thermal thermal_zone0: failed to read out thermal zone (-110)
[19343.319587] thermal thermal_zone0: failed to read out thermal zone (-110)
[19345.479573] thermal thermal_zone0: failed to read out thermal zone (-110)
[19347.639566] thermal thermal_zone0: failed to read out thermal zone (-110)
[19349.731586] systemd[1]: systemd-logind.service stop-sigterm timed out. Killing.
[19349.799550] thermal thermal_zone0: failed to read out thermal zone (-110)
[19351.959542] thermal thermal_zone0: failed to read out thermal zone (-110)
[19354.119522] thermal thermal_zone0: failed to read out thermal zone (-110)
[19356.279502] thermal thermal_zone0: failed to read out thermal zone (-110)
[19358.439499] thermal thermal_zone0: failed to read out thermal zone (-110)
[19360.599479] thermal thermal_zone0: failed to read out thermal zone (-110)
[19362.759476] thermal thermal_zone0: failed to read out thermal zone (-110)
[19364.919499] thermal thermal_zone0: failed to read out thermal zone (-110)
[19367.079435] thermal thermal_zone0: failed to read out thermal zone (-110)
[19369.239427] thermal thermal_zone0: failed to read out thermal zone (-110)
[19371.399444] thermal thermal_zone0: failed to read out thermal zone (-110)
[19373.559402] thermal thermal_zone0: failed to read out thermal zone (-110)
[19375.719385] thermal thermal_zone0: failed to read out thermal zone (-110)
[19375.749293] INFO: rcu_sched detected stalls on CPUs/tasks:
[19375.754811] 	0-...: (1 GPs behind) idle=116/140000000000000/0 softirq=211124/211128 fqs=6102 
[19375.763334] 	(detected by 1, t=14712 jiffies, g=119264, c=119263, q=3721)
[19375.770132] Sending NMI from CPU 1 to CPUs 0:

 

0
532910

532910

Posted
Posted

I confirm this bug. After updating orange pi zero to 4.14.15-sunxi it hangs occasionally.

 

Feb  4 06:32:27 localhost kernel: [24810.496488] NMI backtrace for cpu 0
Feb  4 06:32:27 localhost kernel: [24810.496498] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.14.15-sunxi #28
Feb  4 06:32:27 localhost kernel: [24810.496501] Hardware name: Allwinner sun8i Family
Feb  4 06:32:27 localhost kernel: [24810.496506] task: c0d07780 task.stack: c0d00000
Feb  4 06:32:27 localhost kernel: [24810.496523] PC is at arch_cpu_idle+0x28/0x2c
Feb  4 06:32:27 localhost kernel: [24810.496529] LR is at arch_cpu_idle+0x27/0x2c
Feb  4 06:32:27 localhost kernel: [24810.496533] pc : [<c01070e8>]    lr : [<c01070e7>]    psr: 40070033
Feb  4 06:32:27 localhost kernel: [24810.496536] sp : c0d01f80  ip : 00000018  fp : c0c5ea30
Feb  4 06:32:27 localhost kernel: [24810.496539] r10: 00000000  r9 : 00000000  r8 : c0ddd8eb
Feb  4 06:32:27 localhost kernel: [24810.496544] r7 : c0cb6438  r6 : c0d03f6c  r5 : c0d03fcc  r4 : ffffe000
Feb  4 06:32:27 localhost kernel: [24810.496547] r3 : c0116561  r2 : 00000000  r1 : 00000000  r0 : 00000001
Feb  4 06:32:27 localhost kernel: [24810.496554] Flags: nZcv  IRQs on  FIQs on  Mode SVC_32  ISA Thumb  Segment none
Feb  4 06:32:27 localhost kernel: [24810.496558] Control: 50c5387d  Table: 5a21006a  DAC: 00000051
Feb  4 06:32:27 localhost kernel: [24810.496564] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.14.15-sunxi #28
Feb  4 06:32:27 localhost kernel: [24810.496566] Hardware name: Allwinner sun8i Family
Feb  4 06:32:27 localhost kernel: [24810.496592] [<c010db15>] (unwind_backtrace) from [<c010a0d9>] (show_stack+0x11/0x1
4)
Feb  4 06:32:27 localhost kernel: [24810.496605] [<c010a0d9>] (show_stack) from [<c0867e49>] (dump_stack+0x69/0x78)
Feb  4 06:32:27 localhost kernel: [24810.496621] [<c0867e49>] (dump_stack) from [<c086bc0f>] (nmi_cpu_backtrace+0x8b/0x
d4)
Feb  4 06:32:27 localhost kernel: [24810.496634] [<c086bc0f>] (nmi_cpu_backtrace) from [<c010ca01>] (handle_IPI+0x75/0x
278)
Feb  4 06:32:27 localhost kernel: [24810.496645] [<c010ca01>] (handle_IPI) from [<c01013e3>] (gic_handle_irq+0x67/0x68)
Feb  4 06:32:27 localhost kernel: [24810.496654] [<c01013e3>] (gic_handle_irq) from [<c010aa25>] (__irq_svc+0x65/0x94)
Feb  4 06:32:27 localhost kernel: [24810.496659] Exception stack(0xc0d01f30 to 0xc0d01f78)
Feb  4 06:32:27 localhost kernel: [24810.496666] 1f20:                                     00000001 00000000 00000000 c
0116561
Feb  4 06:32:27 localhost kernel: [24810.496675] 1f40: ffffe000 c0d03fcc c0d03f6c c0cb6438 c0ddd8eb 00000000 00000000 c
0c5ea30
Feb  4 06:32:27 localhost kernel: [24810.496682] 1f60: 00000018 c0d01f80 c01070e7 c01070e8 40070033 ffffffff
Feb  4 06:32:27 localhost kernel: [24810.496693] [<c010aa25>] (__irq_svc) from [<c01070e8>] (arch_cpu_idle+0x28/0x2c)
Feb  4 06:32:27 localhost kernel: [24810.496708] [<c01070e8>] (arch_cpu_idle) from [<c014c93d>] (do_idle+0x115/0x16c)
Feb  4 06:32:27 localhost kernel: [24810.496721] [<c014c93d>] (do_idle) from [<c014cb89>] (cpu_startup_entry+0x19/0x1c)
Feb  4 06:32:27 localhost kernel: [24810.496733] [<c014cb89>] (cpu_startup_entry) from [<c0c00ad5>] (start_kernel+0x353
/0x36a)
Feb  4 06:32:27 localhost kernel: [24810.497496] rcu_sched kthread starved for 6304 jiffies! g12210 c12209 f0x0 RCU_GP_
WAIT_FQS(3) ->state=0x402 ->cpu=0
Feb  4 06:32:27 localhost kernel: [24810.497502] rcu_sched       I    0     8      2 0x00000000
Feb  4 06:32:27 localhost kernel: [24810.497523] [<c08762d9>] (__schedule) from [<c087684f>] (schedule+0x2f/0x68)
Feb  4 06:32:27 localhost kernel: [24810.497538] [<c087684f>] (schedule) from [<c08790ed>] (schedule_timeout+0x75/0x314
)
Feb  4 06:32:27 localhost kernel: [24810.497555] [<c08790ed>] (schedule_timeout) from [<c0164f95>] (rcu_gp_kthread+0x41
5/0x674)
Feb  4 06:32:27 localhost kernel: [24810.497572] [<c0164f95>] (rcu_gp_kthread) from [<c0131a9d>] (kthread+0xfd/0x104)
Feb  4 06:32:27 localhost kernel: [24810.497587] [<c0131a9d>] (kthread) from [<c0106719>] (ret_from_fork+0x11/0x38)
Feb  4 06:32:50 localhost kernel: [24833.077001] INFO: rcu_sched detected stalls on CPUs/tasks:
Feb  4 06:32:50 localhost kernel: [24833.077650]        0-...: (0 ticks this GP) idle=252/140000000000000/0 softirq=212
452/212452 fqs=0
Feb  4 06:32:50 localhost kernel: [24833.077743]        (detected by 3, t=2102 jiffies, g=12216, c=12215, q=2956)
Feb  4 06:32:50 localhost kernel: [24833.077921] Sending NMI from CPU 3 to CPUs 0:

0
Wladimir Mutel

Wladimir Mutel

Posted
Posted (edited)

I had similar experience on my Cubietruck A20 (rcu_sched stalls and rejected forks)

It happened after my first reboot with kernel 4.14.15-sunxi (after some uptime with kernel 4.13.16)
I just run 'sudo cruft' at that time
  

Feb  2 15:58:35 localhost kernel: [102644.225181] INFO: rcu_sched detected stalls on CPUs/tasks:
Feb  2 15:58:35 localhost kernel: [102644.225228]       1-...: (1 GPs behind) idle=272/140000000000000/0 softirq=1869227/1869228 fqs=1050
Feb  2 15:58:35 localhost kernel: [102644.225232]       (detected by 0, t=2102 jiffies, g=798178, c=798177, q=5594)
Feb  2 15:58:35 localhost kernel: [102644.225258] Sending NMI from CPU 0 to CPUs 1:
Feb  2 15:59:38 localhost kernel: [102707.275047] INFO: rcu_sched detected stalls on CPUs/tasks:
Feb  2 15:59:38 localhost kernel: [102707.275093]       1-...: (1 GPs behind) idle=272/140000000000000/0 softirq=1869227/1869228 fqs=3702
Feb  2 15:59:38 localhost kernel: [102707.275097]       (detected by 0, t=8407 jiffies, g=798178, c=798177, q=68757)
Feb  2 15:59:38 localhost kernel: [102707.275122] Sending NMI from CPU 0 to CPUs 1:
Feb  2 15:59:51 localhost kernel: [102730.056354] cgroup: fork rejected by pids controller in /user.slice/user-1001.slice/session-222.scope
Feb  2 16:00:41 localhost kernel: [102770.324854] INFO: rcu_sched detected stalls on CPUs/tasks:
Feb  2 16:00:41 localhost kernel: [102770.324890]       1-...: (1 GPs behind) idle=272/140000000000000/0 softirq=1869227/1869228 fqs=6354
Feb  2 16:00:41 localhost kernel: [102770.324894]       (detected by 0, t=14712 jiffies, g=798178, c=798177, q=89964)
Feb  2 16:00:41 localhost kernel: [102770.324918] Sending NMI from CPU 0 to CPUs 1:
Feb  2 16:01:45 localhost kernel: [102833.374711] INFO: rcu_sched detected stalls on CPUs/tasks:
Feb  2 16:01:45 localhost kernel: [102833.374751]       1-...: (1 GPs behind) idle=272/140000000000000/0 softirq=1869227/1869228 fqs=9006
Feb  2 16:01:45 localhost kernel: [102833.374756]       (detected by 0, t=21017 jiffies, g=798178, c=798177, q=90505)
Feb  2 16:01:45 localhost kernel: [102833.374781] Sending NMI from CPU 0 to CPUs 1:
Feb  2 16:02:48 localhost kernel: [102896.424568] INFO: rcu_sched detected stalls on CPUs/tasks:
Feb  2 16:02:48 localhost kernel: [102896.424608]       1-...: (1 GPs behind) idle=272/140000000000000/0 softirq=1869227/1869228 fqs=11657
Feb  2 16:02:48 localhost kernel: [102896.424612]       (detected by 0, t=27322 jiffies, g=798178, c=798177, q=91242)
Feb  2 16:02:48 localhost kernel: [102896.424636] Sending NMI from CPU 0 to CPUs 1:
Feb  2 16:03:51 localhost kernel: [102959.474426] INFO: rcu_sched detected stalls on CPUs/tasks:
Feb  2 16:03:51 localhost kernel: [102959.474465]       1-...: (1 GPs behind) idle=272/140000000000000/0 softirq=1869227/1869228 fqs=14307
Feb  2 16:03:51 localhost kernel: [102959.474469]       (detected by 0, t=33627 jiffies, g=798178, c=798177, q=92168)
Feb  2 16:03:51 localhost kernel: [102959.474493] Sending NMI from CPU 0 to CPUs 1:
Feb  2 16:04:31 localhost kernel: [103009.885942] cgroup: fork rejected by pids controller in /user.slice/user-1001.slice/session-229.scope

Then I rebooted the board locally by its power switch, and nothing similar happened after that

Edited by Wladimir Mutel
0
René Kliment

René Kliment

Posted
  • Author
Posted

The only workaround I have been able to come up so far is capping the CPU frequency pretty low - 480 MHz max - bigger wouldn't work for me and would always stall after a couple of hours.
I've been able to run it for 2 days straight now with  

cpufreq-set --max 500Mhz

 

0
Igor

Igor

Posted
Posted

It seems that latest nightly build (4.14.18-sunxi/4.14.18-sunxi64) also works just fine. I am currently heavily stressing out:

- Cubietruck
- Orange Prime
- Orange 2E

Uptime: 15 hours 56 minutes 50 seconds

If they are still up by the end of the day, I'll push kernel to the repository.

0
René Kliment

René Kliment

Posted
  • Author
Posted
  On 2/10/2018 at 7:38 AM, Igor said:

It seems that latest nightly build (4.14.18-sunxi/4.14.18-sunxi64) also works just fine. I am currently heavily stressing out:

- Cubietruck
- Orange Prime
- Orange 2E

Uptime: 15 hours 56 minutes 50 seconds

If they are still up by the end of the day, I'll push kernel to the repository.

Expand  

Yes, it seems that .17 and .18 work fine. Thanks!

1
René Kliment

René Kliment

Posted
  • Author
Posted
  On 7/17/2018 at 6:29 PM, jonik said:

Same crashes with Orange pi pc 

Linux orangepipc 4.14.18-sunxi #24 SMP Fri Feb 9 16:24:32 CET 2018 armv7l GNU/Linux
made a cpufreq-set --max 500Mhz

I will see for how long it wont crash

 

serial_console.logFetching info...

Expand  

 

Yes, I can confirm I had freezes on my OPi +2e with a few latest 4.14.x kernels and I had to cap the CPU freq to 648 MHz - anything above would lead to a system freeze. I don't have a log unfortunately, but I believe it would be same as yours.

I see there is kernel 4.17.6 available, testing it now to see if it is fixed or not.
Maybe a patch that fixed this before got left out?
@Igor

0
Igor

Igor

Posted
Posted
  On 7/18/2018 at 7:25 AM, René Kliment said:

Maybe a patch that fixed this before got left out?

Expand  


It's also possible that some boards are much lower quality than expected with our already conservative settings. It's not just kernel, but also u-boot. Both are changed now.

0
René Kliment

René Kliment

Posted
  • Author
Posted
  On 7/18/2018 at 8:40 AM, Igor said:


It's also possible that some boards are much lower quality than expected with our already conservative settings. It's not just kernel, but also u-boot. Both are changed now.

Expand  

 

Board quality is definitely a factor. However, it worked fine for me in early 4.14 kernels, then got broken, then got fixed again and now it's broken again so it seems like a kernel and/or uboot issue. Let's hope it will magically fix itself again :-O

I can confirm that it froze with 4.17.6. I've upgraded to 4.17.8 and connected the serial console to catch the dmesg output when it freezes again.

0
Links2004

Links2004

Posted
Posted

 

 

Hi,

 

can confirm that this still is a big Issue, (running a Orange PI PC)

since I updatet from 4.14.18-sunxi to 4.17.7-sunxi the system is in a dead state very.

before the update the system where running for month stable.

 

Interesting is that the first notable think is that ssh stops working (aktive session is dead and new session gets a timeout),

ping and the home automation stuff running there will run ~1hour more and then the system is dead complete.

 

some log which may or may not helps here:

 

  Reveal hidden contents
  Reveal hidden contents

 

 the last log entitys are spammed with rtc errors.

 

have changed the /etc/default/cpufrequtils to:

 

  Reveal hidden contents


since lower speed may helps.

 

0
René Kliment

René Kliment

Posted
  • Author
Posted

Yeah, so can confirm the thing with my log too.

  

[314330.237556] INFO: rcu_sched detected stalls on CPUs/tasks:
[314330.243170]         (detected by 2, t=107378770 jiffies, g=3815634, c=3815633, q=30640)
[314330.250673] All QSes seen, last rcu_sched kthread activity 107378777 (421779489-314400712), jiffies_till_next_fqs=3, root ->qsmask 0x0
[314330.263135] rcu_sched kthread starved for 107378790 jiffies! g3815634 c3815633 f0x2 RCU_GP_WAIT_FQS(3) ->state=0x0 ->cpu=2
[314330.274266] RCU grace-period kthread stack dump:
[314330.237556] INFO: rcu_sched detected stalls on CPUs/tasks:
[314330.243170]         (detected by 2, t=107378770 jiffies, g=3815634, c=3815633, q=30640)
[314330.250673] All QSes seen, last rcu_sched kthread activity 107378777 (421779489-314400712), jiffies_till_next_fqs=3, root ->qsmask 0x0
[314330.262839] mysqld          R  running task        0  2081   1613 0x00000002
[314330.262877] [<c010cef9>] (unwind_backtrace) from [<c010a471>] (show_stack+0x11/0x14)
[314330.262892] [<c010a471>] (show_stack) from [<c0166fbd>] (rcu_check_callbacks+0x62d/0x630)
[314330.262905] [<c0166fbd>] (rcu_check_callbacks) from [<c016b063>] (update_process_times+0x2b/0x48)
[314330.262918] [<c016b063>] (update_process_times) from [<c01784eb>] (tick_sched_timer+0x37/0x74)
[314330.262930] [<c01784eb>] (tick_sched_timer) from [<c016b89b>] (__hrtimer_run_queues+0xff/0x214)
[314330.262939] [<c016b89b>] (__hrtimer_run_queues) from [<c016c321>] (hrtimer_interrupt+0xb5/0x1fc)
[314330.262954] [<c016c321>] (hrtimer_interrupt) from [<c06cc119>] (arch_timer_handler_phys+0x25/0x28)
[314330.262968] [<c06cc119>] (arch_timer_handler_phys) from [<c015ea3f>] (handle_percpu_devid_irq+0x5f/0x19c)
[314330.262978] [<c015ea3f>] (handle_percpu_devid_irq) from [<c015b13d>] (generic_handle_irq+0x1d/0x28)
[314330.262988] [<c015b13d>] (generic_handle_irq) from [<c015b5dd>] (__handle_domain_irq+0x45/0x84)
[314330.263001] [<c015b5dd>] (__handle_domain_irq) from [<c04dacb7>] (gic_handle_irq+0x43/0x74)
[314330.263012] [<c04dacb7>] (gic_handle_irq) from [<c0101a25>] (__irq_svc+0x65/0x94)
[314330.263017] Exception stack(0xe7347e40 to 0xe7347e88)
[314330.263025] 7e40: 00000000 00000000 ffff0001 e7c9b280 e7c9b210 e7347f30 e7347f18 00000000
[314330.263032] 7e60: e9a3e840 ee273000 e7c9b280 e7c9b080 e7c9b210 e7347e90 c03c0a87 c0154b28
[314330.263037] 7e80: 60080033 ffffffff
[314330.263050] [<c0101a25>] (__irq_svc) from [<c0154b28>] (down_write_trylock+0x28/0x48)
[314330.263063] [<c0154b28>] (down_write_trylock) from [<c03c0a87>] (btrfs_file_write_iter+0x5f/0x4d0)
[314330.263078] [<c03c0a87>] (btrfs_file_write_iter) from [<c021d9e5>] (new_sync_write+0x7d/0xa0)
[314330.263088] [<c021d9e5>] (new_sync_write) from [<c021f547>] (vfs_write+0x77/0x144)
[314330.263096] [<c021f547>] (vfs_write) from [<c021f7df>] (ksys_pwrite64+0x4b/0x5c)
[314330.263104] [<c021f7df>] (ksys_pwrite64) from [<c0101001>] (ret_fast_syscall+0x1/0x62)
[314330.263107] Exception stack(0xe7347fa8 to 0xe7347ff0)
[314330.263113] 7fa0:                   00100000 00000000 00000003 a5004000 00004000 00000000
[314330.263121] 7fc0: 00100000 00000000 00004000 000000b5 00000003 00000000 00a8990c a5004000
[314330.263126] 7fe0: 000000b5 a47fe8f8 b6ed419b b6ed5456
[314330.263135] rcu_sched kthread starved for 107378790 jiffies! g3815634 c3815633 f0x2 RCU_GP_WAIT_FQS(3) ->state=0x0 ->cpu=2
[314330.274266] RCU grace-period kthread stack dump:
[314330.278970] rcu_sched       R  running task        0     9      2 0x00000000
[314330.278994] [<c080e9c7>] (__schedule) from [<c080ef2b>] (schedule+0x2f/0x68)
[314330.279004] [<c080ef2b>] (schedule) from [<c08114e1>] (schedule_timeout+0x69/0x2e4)
[314330.279017] [<c08114e1>] (schedule_timeout) from [<c0165869>] (rcu_gp_kthread+0x405/0x638)
[314330.279028] [<c0165869>] (rcu_gp_kthread) from [<c0130569>] (kthread+0xfd/0x104)
[314330.279036] [<c0130569>] (kthread) from [<c01010f9>] (ret_from_fork+0x11/0x38)
[314330.279040] Exception stack(0xee917fb0 to 0xee917ff8)
[314330.279045] 7fa0:                                     00000000 00000000 00000000 00000000
[314330.279051] 7fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[314330.279057] 7fe0: 00000000 00000000 00000000 00000000 00000013 00000000

 

0
AndrewT

AndrewT

Posted
Posted

Same problem here:

 

Banana PI M1 - ARMBIAN 5.38 stable Debian GNU/Linux 9 (stretch) 4.14.18-sunxi

 

It makes a daily reboot and runs for a couple of days, maybe 3 to 5.

Then the system partial stops to work.

SSH login works, but my cronjobs don't run.

And it seems that the system is just using one of the two cpu cores.

 

When I try to restart the system with "reboot", it doesn't come up again.

I have to use the reset-button.

 

Here is my log: 

Aug 12 21:51:26 pi-router kernel: [82260.470855] INFO: rcu_sched detected stalls on CPUs/tasks:
Aug 12 21:51:26 pi-router kernel: [82260.470892]        1-...: (1 GPs behind) idle=fe6/140000000000000/0 softirq=172179/172185 fqs=639049
Aug 12 21:51:26 pi-router kernel: [82260.470897]        (detected by 0, t=1578352 jiffies, g=83018, c=83017, q=198500)
Aug 12 21:51:26 pi-router kernel: [82260.470922] Sending NMI from CPU 0 to CPUs 1:
Aug 12 21:51:27 pi-router kernel: [82271.651723] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:51:29 pi-router kernel: [82273.811868] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:51:31 pi-router kernel: [82275.972016] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:51:34 pi-router kernel: [82278.132167] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:51:36 pi-router kernel: [82280.292319] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:51:38 pi-router kernel: [82282.452471] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:51:40 pi-router kernel: [82284.612622] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:51:42 pi-router kernel: [82286.772783] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:51:44 pi-router kernel: [82288.932919] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:51:46 pi-router kernel: [82291.093111] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:51:49 pi-router kernel: [82293.253221] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:51:51 pi-router kernel: [82295.413375] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:51:53 pi-router kernel: [82297.573525] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:51:55 pi-router kernel: [82299.733682] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:51:57 pi-router kernel: [82301.893902] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:51:59 pi-router kernel: [82304.053975] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:02 pi-router kernel: [82306.214126] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:04 pi-router kernel: [82308.374273] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:06 pi-router kernel: [82310.534438] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:08 pi-router kernel: [82312.694576] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:10 pi-router kernel: [82314.854736] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:12 pi-router kernel: [82317.015819] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:15 pi-router kernel: [82319.175042] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:17 pi-router kernel: [82321.335187] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:19 pi-router kernel: [82323.495331] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:29 pi-router kernel: [82323.525252] INFO: rcu_sched detected stalls on CPUs/tasks:
Aug 12 21:52:29 pi-router kernel: [82323.525290]        1-...: (1 GPs behind) idle=fe6/140000000000000/0 softirq=172179/172185 fqs=641602
Aug 12 21:52:29 pi-router kernel: [82323.525295]        (detected by 0, t=1584657 jiffies, g=83018, c=83017, q=199148)
Aug 12 21:52:29 pi-router kernel: [82323.525319] Sending NMI from CPU 0 to CPUs 1:
Aug 12 21:52:30 pi-router kernel: [82334.696137] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:32 pi-router kernel: [82336.856262] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:34 pi-router kernel: [82339.016434] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:37 pi-router kernel: [82341.176561] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:39 pi-router kernel: [82343.336713] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:41 pi-router kernel: [82345.496864] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:43 pi-router kernel: [82347.657017] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:45 pi-router kernel: [82349.817192] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:47 pi-router kernel: [82351.977312] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:50 pi-router kernel: [82354.137477] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:52 pi-router kernel: [82356.297613] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:54 pi-router kernel: [82358.457766] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:56 pi-router kernel: [82360.617913] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:52:58 pi-router kernel: [82362.778074] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:53:00 pi-router kernel: [82364.938289] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:53:02 pi-router kernel: [82367.098371] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:53:05 pi-router kernel: [82369.258521] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:53:07 pi-router kernel: [82371.418683] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:53:09 pi-router kernel: [82373.578825] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:53:11 pi-router kernel: [82375.738982] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:53:13 pi-router kernel: [82377.899121] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:53:15 pi-router kernel: [82380.060167] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:53:18 pi-router kernel: [82382.219429] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:53:20 pi-router kernel: [82384.379579] thermal thermal_zone0: failed to read out thermal zone (-110)
Aug 12 21:53:22 pi-router kernel: [82386.539739] thermal thermal_zone0: failed to read out thermal zone (-110)

 

0
menno

menno

Posted
Posted
  On 8/24/2018 at 12:09 PM, René Kliment said:

I've been running 4.17.14-sunxi for 4 days straight now and it's crunching two `openssl speed` in a loop at 1200 MHz. So far so good.

Maybe this patch has something to do with it? https://github.com/armbian/build/commit/3326ccc11648e5ff482102ec401b22cc795006ae

I don't have time to compare with and without, but I'm super glad it works now :-)

Expand  

I was having very similar issues with my Orange Pi+ 2E and even with all patches applied (I'm running megous' mainline kernel[1] that includes the patch you linked) and I was still having trouble with random hangs and rcu stalls. I was able to get the machine stable by setting cpufreq to a fixed frequency. It does not matter if I set it to low or high so mine's now stable running at 1.37Ghz for days.

 

Did you lock yours at 1200Mhz or was it just never idle ? (Is it still stable for you?)

 

[1] https://github.com/megous/linux/

0
piknew

piknew

Posted
Posted
  On 9/30/2018 at 12:20 PM, menno said:

I was able to get the machine stable by setting cpufreq to a fixed frequency. It does not matter if I set it to low or high so mine's now stable running at 1.37Ghz for days.

Expand  

 

Could you please point out how to achieve this? All my H3 (also H2+) devices suffer from the issue as described. Even with newest kernel. So, after reading I decided to test stability with "constant cpu frequency".

 

EDIT: I've done it by:

 

[root@PKTEST ~]# cat /etc/default/cpufrequtils
# WARNING: this file will be replaced on board support package (linux-root-...) upgrade
ENABLE=true
MIN_SPEED=408000
MAX_SPEED=1296000
#GOVERNOR=ondemand
GOVERNOR=performance
[root@PKTEST ~]#

 

Question: would it be possible not to overwrite cpufrequtils file once support package is installed/upgraded?

0
piknew

piknew

Posted
Posted

Could you please take a look here:

 

Considering that when running on performance mode - system seems to be stable but once switched to "ondemand" - unstable. May the "mismatch" be the root cause here?

 

0
Igor

Igor

Posted
Posted
  On 10/21/2018 at 1:11 PM, piknew said:

May the "mismatch" be the root cause here?

Expand  


Yes it can be. It was noticed on pure mainline builds as well.

 

  On 10/10/2018 at 5:14 PM, piknew said:

Question: would it be possible not to overwrite cpufrequtils file once support package is installed/upgraded?

Expand  


This part is due to rework but since developers count is low, it takes a lot of time to get there.

0
piknew

piknew

Posted
Posted

For now I have modified cpufrequtils as following for my three boards (orangepipc, orangepiplus 2 GB, orangepiplus2e): 

[root@PKBACKUP ~]# cat /etc/default/cpufrequtils
# WARNING: this file will be replaced on board support package (linux-root-...) upgrade
ENABLE=true
MIN_SPEED=480000
MAX_SPEED=1296000
GOVERNOR=ondemand
#GOVERNOR=performance

 

For orangepizero as following: 

[root@PKOTHER ~]# cat /etc/default/cpufrequtils
# WARNING: this file will be replaced on board support package (linux-root-...) upgrade
ENABLE=true
MIN_SPEED=240000
MAX_SPEED=1200000
GOVERNOR=ondemand
#GOVERNOR=performance

 

Only orangepiplus2e is somehow loaded. With my own backup software, which is run twice a day - but in the past it was enough to freeze the board after a few days.

 

I will update if the change helped or not.

 

Question: I have noticed that orangepizero is using lower "MAX" settings. I understand as this device had an issue with overheating. How about other platforms, why lowest frequency settings is 480 MHz, not 240 MHz (which is also supported by H3)?

 

Some results for my platforms (please note that 816 MHz is common lowest choice for H3):

 

orangepiplu2e: 

[root@PKBACKUP ~]# armbianmonitor -m
Stop monitoring using [ctrl]-[c]
Time        CPU    load %cpu %sys %usr %nice %io %irq   CPU  C.St.

16:36:53: 1296MHz  0.00   6%   1%   2%   0%   1%   0% 42.5°C  0/9
16:36:58:  816MHz  0.00   0%   0%   0%   0%   0%   0% 43.0°C  0/9
16:37:03:  816MHz  0.00   0%   0%   0%   0%   0%   0% 42.8°C  0/9
16:37:08:  816MHz  0.00   0%   0%   0%   0%   0%   0% 42.8°C  0/9
16:37:14:  816MHz  0.00   0%   0%   0%   0%   0%   0% 40.2°C  0/9
16:37:19:  816MHz  0.00   0%   0%   0%   0%   0%   0% 43.4°C  0/9
16:37:24:  816MHz  0.00   0%   0%   0%   0%   0%   0% 41.7°C  0/9
16:37:29:  816MHz  0.00   0%   0%   0%   0%   0%   0% 41.6°C  0/9
16:37:34:  816MHz  0.00   0%   0%   0%   0%   0%   0% 42.5°C  0/9^C

 

orangepiplus: 

[root@PKHELPER ~]# armbianmonitor -m
Stop monitoring using [ctrl]-[c]
Time        CPU    load %cpu %sys %usr %nice %io %irq   CPU  C.St.

16:39:34:  816MHz  0.13   0%   0%   0%   0%   0%   0% 44.2°C  0/9
16:39:39: 1296MHz  0.18   0%   0%   0%   0%   0%   0% 43.8°C  0/9
16:39:44:  816MHz  0.17   0%   0%   0%   0%   0%   0% 43.9°C  0/9
16:39:49: 1296MHz  0.16   0%   0%   0%   0%   0%   0% 43.7°C  0/9
16:39:54:  816MHz  0.14   0%   0%   0%   0%   0%   0% 44.0°C  0/9
16:39:59: 1296MHz  0.13   0%   0%   0%   0%   0%   0% 44.3°C  0/9
16:40:04:  816MHz  0.12   0%   0%   0%   0%   0%   0% 44.3°C  0/9
16:40:09:  816MHz  0.11   0%   0%   0%   0%   0%   0% 43.7°C  0/9
16:40:15:  816MHz  0.10   0%   0%   0%   0%   0%   0% 43.1°C  0/9
16:40:20:  816MHz  0.09   0%   0%   0%   0%   0%   0% 43.7°C  0/9^C

 

orangepipc: 

[root@PKTEST ~]# armbianmonitor -m
Stop monitoring using [ctrl]-[c]
Time        CPU    load %cpu %sys %usr %nice %io %irq   CPU  C.St.

16:41:11: 1296MHz  0.07   0%   0%   0%   0%   0%   0% 35.7°C  0/9
16:41:16:  816MHz  0.06   0%   0%   0%   0%   0%   0% 36.7°C  0/9
16:41:21:  816MHz  0.06   0%   0%   0%   0%   0%   0% 36.8°C  0/9
16:41:26:  816MHz  0.05   0%   0%   0%   0%   0%   0% 35.7°C  0/9
16:41:31:  816MHz  0.05   0%   0%   0%   0%   0%   0% 35.9°C  0/9
16:41:37:  816MHz  0.04   0%   0%   0%   0%   0%   0% 36.5°C  0/9
16:41:42:  816MHz  0.04   0%   0%   0%   0%   0%   0% 35.8°C  0/9
16:41:47:  816MHz  0.04   0%   0%   0%   0%   0%   0% 36.4°C  0/9
16:41:52:  816MHz  0.03   0%   0%   0%   0%   0%   0% 36.2°C  0/9^C

orangepizero: 

[root@PKOTHER ~]# armbianmonitor -m
Stop monitoring using [ctrl]-[c]
Time        CPU    load %cpu %sys %usr %nice %io %irq   CPU  C.St.

16:42:33: 1200MHz  0.00   0%   0%   0%   0%   0%   0% 43.0°C  0/8
16:42:38:  648MHz  0.00   0%   0%   0%   0%   0%   0% 43.1°C  0/8
16:42:43:  648MHz  0.00   0%   0%   0%   0%   0%   0% 43.1°C  0/8
16:42:48:  648MHz  0.00   0%   0%   0%   0%   0%   0% 42.6°C  0/8
16:42:54:  648MHz  0.08   0%   0%   0%   0%   0%   0% 42.5°C  0/8
16:42:59:  648MHz  0.07   0%   0%   0%   0%   0%   0% 43.6°C  0/8
16:43:04:  648MHz  0.06   0%   0%   0%   0%   0%   0% 43.0°C  0/8
16:43:09:  648MHz  0.06   0%   0%   0%   0%   0%   0% 42.6°C  0/8^C

 

 

0
piknew

piknew

Posted
Posted

After correcting cpufrequtils file - no issues so far:

  

[root@PKBACKUP ~]# date
Fri Oct 26 11:03:47 CEST 2018
[root@PKBACKUP ~]# uptime
 11:03:53 up 4 days, 16:14,  1 user,  load average: 0.00, 0.00, 0.00
[root@PKBACKUP ~]# ll /etc/default/cpufrequtils
-r--r--r-- 1 root root 175 Oct 18 18:01 /etc/default/cpufrequtils
[root@PKBACKUP ~]# cat /etc/default/cpufrequtils
# WARNING: this file will be replaced on board support package (linux-root-...) upgrade
ENABLE=true
MIN_SPEED=480000
MAX_SPEED=1296000
GOVERNOR=ondemand
#GOVERNOR=performance
[root@PKBACKUP ~]#

Anybody else (who's SBC was impacted by this issue) can verify the same?

0
piknew

piknew

Posted
Posted

Still OK (I can confirm this for all of my "orange" SBCs):

  

[root@PKBACKUP ~]# date
Mon Oct 29 20:21:27 CET 2018
[root@PKBACKUP ~]# uptime
 20:21:32 up 8 days,  2:32,  1 user,  load average: 0.10, 0.03, 0.01
[root@PKBACKUP ~]# ll /etc/default/cpufrequtils
-r--r--r-- 1 root root 175 Oct 18 18:01 /etc/default/cpufrequtils
[root@PKBACKUP ~]# cat /etc/default/cpufrequtils
# WARNING: this file will be replaced on board support package (linux-root-...) upgrade
ENABLE=true
MIN_SPEED=480000
MAX_SPEED=1296000
GOVERNOR=ondemand
#GOVERNOR=performance
[root@PKBACKUP ~]#

 

0
sunzone

sunzone

Posted
Posted (edited)

Unfortunately I gets the same "rcu_sched detected stalls on CPUs/tasks" error even with corrected "/etc/default/cpufrequtils" file

 

  Reveal hidden contents


This kernel hang occurred right after 2 CRON jobs started at the same time, hence CPU was not in idle at the time.

/etc/default/cpufrequtils  

# WARNING: this file will be replaced on board support package (linux-root-...) upgrade
ENABLE=true
MIN_SPEED=480000
MAX_SPEED=816000
GOVERNOR=ondemand

 

You can see armbian -U output here:  

https://paste.c-net.org/b1965858-723d-3409-2e8e-264e9e05d156

 

Running: 4.19.62-sunxi on Orange Pi Zero.

 

I am going to check with GOVERNOR=performance

Can anybody give me a hint/clues what might have caused this error and how to solve it?

 

Thank you

Edited by TRS-80
put long output inside spoiler
0
Marmelade

Marmelade

Posted
Posted

Armbian 21.08.1 Focal

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.2 LTS"

 

5.10.60-sunxi

Gets frozen after 10 min - 10h, while I run a bash script calling rather simple program written in C.

This GPS C program just reads GPS at /dev/ttyS2 at 9600bd

I have made a lot of work with the script and GPS-program having no luck but learned a lot.

 

Processor speed 480-1102MHz.

I took a photo of the frozen screen and made Google find this forum.

 

Here is the part of my script, which gets data  from the GPS-program:

 

timeout 3 /home/bin/i2cLCD/gps >$TMPFILE &
pid=$!
echo "PID: $pid"
wait $pid
read uline < $TMPFILE

 

0

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
 Share
Go to topic listing
×
×
  • Create New...

Important Information

Terms of Use - Privacy Policy - Guidelines

  I accept