René Kliment

I've been running 4.17.14-sunxi for 4 days straight now and it's crunching two `openssl speed` in a loop at 1200 MHz. So far so good. Maybe this patch has something to do with it? https://github.com/armbian/build/commit/3326ccc11648e5ff482102ec401b22cc795006ae I don't have time to compare with and without, but I'm super glad it works now :-)

Yeah, so can confirm the thing with my log too. [314330.237556] INFO: rcu_sched detected stalls on CPUs/tasks: [314330.243170] (detected by 2, t=107378770 jiffies, g=3815634, c=3815633, q=30640) [314330.250673] All QSes seen, last rcu_sched kthread activity 107378777 (421779489-314400712), jiffies_till_next_fqs=3, root ->qsmask 0x0 [314330.263135] rcu_sched kthread starved for 107378790 jiffies! g3815634 c3815633 f0x2 RCU_GP_WAIT_FQS(3) ->state=0x0 ->cpu=2 [314330.274266] RCU grace-period kthread stack dump: [314330.237556] INFO: rcu_sched detected stalls on CPUs/tasks: [314330.243170] (detected by 2, t=107378770 jiffies, g=3815634, c=3815633, q=30640) [314330.250673] All QSes seen, last rcu_sched kthread activity 107378777 (421779489-314400712), jiffies_till_next_fqs=3, root ->qsmask 0x0 [314330.262839] mysqld R running task 0 2081 1613 0x00000002 [314330.262877] [<c010cef9>] (unwind_backtrace) from [<c010a471>] (show_stack+0x11/0x14) [314330.262892] [<c010a471>] (show_stack) from [<c0166fbd>] (rcu_check_callbacks+0x62d/0x630) [314330.262905] [<c0166fbd>] (rcu_check_callbacks) from [<c016b063>] (update_process_times+0x2b/0x48) [314330.262918] [<c016b063>] (update_process_times) from [<c01784eb>] (tick_sched_timer+0x37/0x74) [314330.262930] [<c01784eb>] (tick_sched_timer) from [<c016b89b>] (__hrtimer_run_queues+0xff/0x214) [314330.262939] [<c016b89b>] (__hrtimer_run_queues) from [<c016c321>] (hrtimer_interrupt+0xb5/0x1fc) [314330.262954] [<c016c321>] (hrtimer_interrupt) from [<c06cc119>] (arch_timer_handler_phys+0x25/0x28) [314330.262968] [<c06cc119>] (arch_timer_handler_phys) from [<c015ea3f>] (handle_percpu_devid_irq+0x5f/0x19c) [314330.262978] [<c015ea3f>] (handle_percpu_devid_irq) from [<c015b13d>] (generic_handle_irq+0x1d/0x28) [314330.262988] [<c015b13d>] (generic_handle_irq) from [<c015b5dd>] (__handle_domain_irq+0x45/0x84) [314330.263001] [<c015b5dd>] (__handle_domain_irq) from [<c04dacb7>] (gic_handle_irq+0x43/0x74) [314330.263012] [<c04dacb7>] (gic_handle_irq) from [<c0101a25>] (__irq_svc+0x65/0x94) [314330.263017] Exception stack(0xe7347e40 to 0xe7347e88) [314330.263025] 7e40: 00000000 00000000 ffff0001 e7c9b280 e7c9b210 e7347f30 e7347f18 00000000 [314330.263032] 7e60: e9a3e840 ee273000 e7c9b280 e7c9b080 e7c9b210 e7347e90 c03c0a87 c0154b28 [314330.263037] 7e80: 60080033 ffffffff [314330.263050] [<c0101a25>] (__irq_svc) from [<c0154b28>] (down_write_trylock+0x28/0x48) [314330.263063] [<c0154b28>] (down_write_trylock) from [<c03c0a87>] (btrfs_file_write_iter+0x5f/0x4d0) [314330.263078] [<c03c0a87>] (btrfs_file_write_iter) from [<c021d9e5>] (new_sync_write+0x7d/0xa0) [314330.263088] [<c021d9e5>] (new_sync_write) from [<c021f547>] (vfs_write+0x77/0x144) [314330.263096] [<c021f547>] (vfs_write) from [<c021f7df>] (ksys_pwrite64+0x4b/0x5c) [314330.263104] [<c021f7df>] (ksys_pwrite64) from [<c0101001>] (ret_fast_syscall+0x1/0x62) [314330.263107] Exception stack(0xe7347fa8 to 0xe7347ff0) [314330.263113] 7fa0: 00100000 00000000 00000003 a5004000 00004000 00000000 [314330.263121] 7fc0: 00100000 00000000 00004000 000000b5 00000003 00000000 00a8990c a5004000 [314330.263126] 7fe0: 000000b5 a47fe8f8 b6ed419b b6ed5456 [314330.263135] rcu_sched kthread starved for 107378790 jiffies! g3815634 c3815633 f0x2 RCU_GP_WAIT_FQS(3) ->state=0x0 ->cpu=2 [314330.274266] RCU grace-period kthread stack dump: [314330.278970] rcu_sched R running task 0 9 2 0x00000000 [314330.278994] [<c080e9c7>] (__schedule) from [<c080ef2b>] (schedule+0x2f/0x68) [314330.279004] [<c080ef2b>] (schedule) from [<c08114e1>] (schedule_timeout+0x69/0x2e4) [314330.279017] [<c08114e1>] (schedule_timeout) from [<c0165869>] (rcu_gp_kthread+0x405/0x638) [314330.279028] [<c0165869>] (rcu_gp_kthread) from [<c0130569>] (kthread+0xfd/0x104) [314330.279036] [<c0130569>] (kthread) from [<c01010f9>] (ret_from_fork+0x11/0x38) [314330.279040] Exception stack(0xee917fb0 to 0xee917ff8) [314330.279045] 7fa0: 00000000 00000000 00000000 00000000 [314330.279051] 7fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [314330.279057] 7fe0: 00000000 00000000 00000000 00000000 00000013 00000000

Board quality is definitely a factor. However, it worked fine for me in early 4.14 kernels, then got broken, then got fixed again and now it's broken again so it seems like a kernel and/or uboot issue. Let's hope it will magically fix itself again :-O I can confirm that it froze with 4.17.6. I've upgraded to 4.17.8 and connected the serial console to catch the dmesg output when it freezes again.

Yes, I can confirm I had freezes on my OPi +2e with a few latest 4.14.x kernels and I had to cap the CPU freq to 648 MHz - anything above would lead to a system freeze. I don't have a log unfortunately, but I believe it would be same as yours. I see there is kernel 4.17.6 available, testing it now to see if it is fixed or not. Maybe a patch that fixed this before got left out? @Igor

This is my very dirty script that seems to work for me: https://gist.github.com/renekliment/707ea4a2dc3f11fc15ed8085f506c57e

Yes, it seems that .17 and .18 work fine. Thanks!

The only workaround I have been able to come up so far is capping the CPU frequency pretty low - 480 MHz max - bigger wouldn't work for me and would always stall after a couple of hours. I've been able to run it for 2 days straight now with cpufreq-set --max 500Mhz

Happens with both idle and busy Pi. I've been trying to rsync+ssh from a remote to an USB connected HDD. The HDD has its own power. Also, the Pi has a heat sink, so no overheating either. This is the weirdest thing ...

Hello everyone. This happens on my OrangePi after a couple of hours of running. I don't think this happened before installing the kernel 4.14. I haven't changed my power supply, which should be working well - custom DC/DC converter connected to GPIO pins. Is anyone else experiencing this? Anything I can do? [19155.760977] INFO: rcu_sched self-detected stall on CPU [19155.760983] INFO: rcu_sched self-detected stall on CPU [19155.760997] INFO: rcu_sched detected stalls on CPUs/tasks: [19155.761010] 2-...: (1 ticks this GP) idle=18a/1/0 softirq=332295/332295 fqs=0 [19155.761017] 0-...: (0 ticks this GP) idle=dbe/1/0 softirq=475103/475103 fqs=0 [19155.761018] [19155.761025] 1-...: (1 GPs behind) idle=f36/1/0 softirq=422883/422884 fqs=0 [19155.761034] 2-...: (1 ticks this GP) idle=18a/1/0 softirq=332295/332295 fqs=0 [19155.761040] rcu_sched kthread starved for 91911 jiffies! g126395 c126394 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x402 ->cpu=3 [19155.761041] [19155.819568] 1-...: (1 GPs behind) idle=f36/1/0 softirq=422883/422884 fqs=0 [19155.826612] (t=91911 jiffies g=126395 c=126394 q=3) [19155.831672] rcu_sched kthread starved for 91911 jiffies! g126395 c126394 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x0 ->cpu=0 [19185.762182] rcu_sched kthread starved for 91911 jiffies! g126395 c126394 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x0 ->cpu=0 [19155.760977] INFO: rcu_sched self-detected stall on CPU [19155.760983] INFO: rcu_sched self-detected stall on CPU [19155.760997] INFO: rcu_sched detected stalls on CPUs/tasks: [19155.761010] 2-...: (1 ticks this GP) idle=18a/1/0 softirq=332295/332295 fqs=0 [19155.761017] 0-...: (0 ticks this GP) idle=dbe/1/0 softirq=475103/475103 fqs=0 [19155.761018] [19155.761025] 1-...: (1 GPs behind) idle=f36/1/0 softirq=422883/422884 fqs=0 [19155.761028] (t=91911 jiffies g=126395 c=126394 q=3) [19155.761034] 2-...: (1 ticks this GP) idle=18a/1/0 softirq=332295/332295 fqs=0 [19155.761040] rcu_sched kthread starved for 91911 jiffies! g126395 c126394 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x402 ->cpu=3 [19155.761041] [19155.761044] rcu_sched I [19155.761048] (detected by 3, t=91911 jiffies, g=126395, c=126394, q=3) [19155.761052] 0 8 2 0x00000000 [19155.761057] Sending NMI from CPU 3 to CPUs 0: [19155.761079] [<c08762b9>] (__schedule) from [<c087682b>] (schedule+0x2f/0x68) [19155.761093] [<c087682b>] (schedule) from [<c087909d>] (schedule_timeout+0x75/0x314) [19155.761108] [<c087909d>] (schedule_timeout) from [<c0164f95>] (rcu_gp_kthread+0x415/0x674) [19155.761123] [<c0164f95>] (rcu_gp_kthread) from [<c0131a9d>] (kthread+0xfd/0x104) [19155.761138] [<c0131a9d>] (kthread) from [<c0106719>] (ret_from_fork+0x11/0x38) [19155.819568] 1-...: (1 GPs behind) idle=f36/1/0 softirq=422883/422884 fqs=0 [19155.826612] (t=91911 jiffies g=126395 c=126394 q=3) [19155.831672] rcu_sched kthread starved for 91911 jiffies! g126395 c126394 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x0 ->cpu=0 [19155.842191] rcu_sched R running task 0 8 2 0x00000000 [19155.842210] [<c08762b9>] (__schedule) from [<c087682b>] (schedule+0x2f/0x68) [19155.842222] [<c087682b>] (schedule) from [<c087909d>] (schedule_timeout+0x75/0x314) [19155.842234] [<c087909d>] (schedule_timeout) from [<c0164f95>] (rcu_gp_kthread+0x415/0x674) [19155.842245] [<c0164f95>] (rcu_gp_kthread) from [<c0131a9d>] (kthread+0xfd/0x104) [19155.842256] [<c0131a9d>] (kthread) from [<c0106719>] (ret_from_fork+0x11/0x38) [19165.761432] Sending NMI from CPU 3 to CPUs 1: [19175.761806] Sending NMI from CPU 3 to CPUs 2: [19185.762182] rcu_sched kthread starved for 91911 jiffies! g126395 c126394 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x0 ->cpu=0 [19185.762186] Sending NMI from CPU 2 to CPUs 0: [19185.772707] rcu_sched R running task 0 8 2 0x00000000 [19185.772726] [<c08762b9>] (__schedule) from [<c087682b>] (schedule+0x2f/0x68) [19185.772738] [<c087682b>] (schedule) from [<c087909d>] (schedule_timeout+0x75/0x314) [19185.772749] [<c087909d>] (schedule_timeout) from [<c0164f95>] (rcu_gp_kthread+0x415/0x674) [19185.772760] [<c0164f95>] (rcu_gp_kthread) from [<c0131a9d>] (kthread+0xfd/0x104) [19185.772772] [<c0131a9d>] (kthread) from [<c0106719>] (ret_from_fork+0x11/0x38) [19195.762559] Sending NMI from CPU 2 to CPUs 1: [19205.762933] NMI backtrace for cpu 2 [19205.762942] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.14.14-sunxi #38 [19205.762945] Hardware name: Allwinner sun8i Family [19205.762959] [<c010db15>] (unwind_backtrace) from [<c010a0d9>] (show_stack+0x11/0x14) [19205.762970] [<c010a0d9>] (show_stack) from [<c0867e29>] (dump_stack+0x69/0x78) [19205.762983] [<c0867e29>] (dump_stack) from [<c086bc37>] (nmi_cpu_backtrace+0xd3/0xd4) [19205.762996] [<c086bc37>] (nmi_cpu_backtrace) from [<c086bccf>] (nmi_trigger_cpumask_backtrace+0x97/0xd0) [19205.763008] [<c086bccf>] (nmi_trigger_cpumask_backtrace) from [<c01662a7>] (rcu_dump_cpu_stacks+0x77/0x94) [19205.763019] [<c01662a7>] (rcu_dump_cpu_stacks) from [<c0165a65>] (rcu_check_callbacks+0x4d5/0x690) [19205.763032] [<c0165a65>] (rcu_check_callbacks) from [<c0169f5f>] (update_process_times+0x2b/0x48) [19205.763046] [<c0169f5f>] (update_process_times) from [<c0177d11>] (tick_sched_timer+0x31/0x68) [19205.763057] [<c0177d11>] (tick_sched_timer) from [<c016ac85>] (__hrtimer_run_queues+0xf5/0x224) [19205.763066] [<c016ac85>] (__hrtimer_run_queues) from [<c016af81>] (hrtimer_interrupt+0x81/0x180) [19205.763079] [<c016af81>] (hrtimer_interrupt) from [<c07483a1>] (arch_timer_handler_phys+0x25/0x28) [19205.763093] [<c07483a1>] (arch_timer_handler_phys) from [<c015d52f>] (handle_percpu_devid_irq+0x57/0x19c) [19205.763107] [<c015d52f>] (handle_percpu_devid_irq) from [<c0159a49>] (generic_handle_irq+0x1d/0x28) [19205.763120] [<c0159a49>] (generic_handle_irq) from [<c0159e59>] (__handle_domain_irq+0x45/0x84) [19205.763131] [<c0159e59>] (__handle_domain_irq) from [<c01013b5>] (gic_handle_irq+0x39/0x68) [19205.763141] [<c01013b5>] (gic_handle_irq) from [<c010aa25>] (__irq_svc+0x65/0x94) [19205.763146] Exception stack(0xee523f78 to 0xee523fc0) [19205.763152] 3f60: 00000001 00000000 [19205.763161] 3f80: 00000000 c0116561 ffffe000 c0d03fcc c0d03f6c c0cb6438 c0ddd8eb 00000000 [19205.763170] 3fa0: 00000000 00000000 00087b5d ee523fc8 c01070e7 c01070e8 40000033 ffffffff [19205.763181] [<c010aa25>] (__irq_svc) from [<c01070e8>] (arch_cpu_idle+0x28/0x2c) [19205.763194] [<c01070e8>] (arch_cpu_idle) from [<c014c93d>] (do_idle+0x115/0x16c) [19205.763206] [<c014c93d>] (do_idle) from [<c014cb89>] (cpu_startup_entry+0x19/0x1c) [19205.763216] [<c014cb89>] (cpu_startup_entry) from [<401016f1>] (0x401016f1) [19205.763226] Sending NMI from CPU 1 to CPUs 0: [19215.763599] NMI backtrace for cpu 1 [19215.763605] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.14.14-sunxi #38 [19215.763608] Hardware name: Allwinner sun8i Family [19215.763618] [<c010db15>] (unwind_backtrace) from [<c010a0d9>] (show_stack+0x11/0x14) [19215.763627] [<c010a0d9>] (show_stack) from [<c0867e29>] (dump_stack+0x69/0x78) [19215.763638] [<c0867e29>] (dump_stack) from [<c086bc37>] (nmi_cpu_backtrace+0xd3/0xd4) [19215.763650] [<c086bc37>] (nmi_cpu_backtrace) from [<c086bccf>] (nmi_trigger_cpumask_backtrace+0x97/0xd0) [19215.763661] [<c086bccf>] (nmi_trigger_cpumask_backtrace) from [<c01662a7>] (rcu_dump_cpu_stacks+0x77/0x94) [19215.763672] [<c01662a7>] (rcu_dump_cpu_stacks) from [<c0165a65>] (rcu_check_callbacks+0x4d5/0x690) [19215.763683] [<c0165a65>] (rcu_check_callbacks) from [<c0169f5f>] (update_process_times+0x2b/0x48) [19215.763695] [<c0169f5f>] (update_process_times) from [<c0177d11>] (tick_sched_timer+0x31/0x68) [19215.763705] [<c0177d11>] (tick_sched_timer) from [<c016ac85>] (__hrtimer_run_queues+0xf5/0x224) [19215.763714] [<c016ac85>] (__hrtimer_run_queues) from [<c016af81>] (hrtimer_interrupt+0x81/0x180) [19215.763724] [<c016af81>] (hrtimer_interrupt) from [<c07483a1>] (arch_timer_handler_phys+0x25/0x28) [19215.763735] [<c07483a1>] (arch_timer_handler_phys) from [<c015d52f>] (handle_percpu_devid_irq+0x57/0x19c) [19215.763746] [<c015d52f>] (handle_percpu_devid_irq) from [<c0159a49>] (generic_handle_irq+0x1d/0x28) [19215.763758] [<c0159a49>] (generic_handle_irq) from [<c0159e59>] (__handle_domain_irq+0x45/0x84) [19215.763768] [<c0159e59>] (__handle_domain_irq) from [<c01013b5>] (gic_handle_irq+0x39/0x68) [19215.763777] [<c01013b5>] (gic_handle_irq) from [<c010aa25>] (__irq_svc+0x65/0x94) [19215.763781] Exception stack(0xee521f78 to 0xee521fc0) [19215.763787] 1f60: 00000001 00000000 [19215.763796] 1f80: 00000000 c0116561 ffffe000 c0d03fcc c0d03f6c c0cb6438 c0ddd8eb 00000000 [19215.763805] 1fa0: 00000000 00000000 00000018 ee521fc8 c01070e7 c01070e8 40000033 ffffffff [19215.763815] [<c010aa25>] (__irq_svc) from [<c01070e8>] (arch_cpu_idle+0x28/0x2c) [19215.763826] [<c01070e8>] (arch_cpu_idle) from [<c014c93d>] (do_idle+0x115/0x16c) [19215.763837] [<c014c93d>] (do_idle) from [<c014cb89>] (cpu_startup_entry+0x19/0x1c) [19215.763846] [<c014cb89>] (cpu_startup_entry) from [<401016f1>] (0x401016f1) [19215.763853] Sending NMI from CPU 1 to CPUs 2: [19225.764386] NMI backtrace for cpu 2 [19225.764393] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.14.14-sunxi #38 [19225.764395] Hardware name: Allwinner sun8i Family [19225.764399] task: ee4f8000 task.stack: ee522000 [19225.764404] PC is at __do_softirq+0x7a/0x25c [19225.764412] LR is at irq_exit+0x7f/0xc4 [19225.764416] pc : [<c0101462>] lr : [<c011ef33>] psr: 40000133 [19225.764419] sp : ee523ee8 ip : 7fffffff fp : 4000001f [19225.764422] r10: c0d02080 r9 : ee434000 r8 : 00000001 [19225.764426] r7 : ffffe000 r6 : 00000282 r5 : 00000000 r4 : ffffe000 [19225.764430] r3 : 00000080 r2 : 00000000 r1 : c0df6740 r0 : c0df6740 [19225.764435] Flags: nZcv IRQs on FIQs on Mode SVC_32 ISA Thumb Segment none [19225.764439] Control: 50c5387d Table: 56ff006a DAC: 00000051 [19225.764444] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.14.14-sunxi #38 [19225.764446] Hardware name: Allwinner sun8i Family [19225.764456] [<c010db15>] (unwind_backtrace) from [<c010a0d9>] (show_stack+0x11/0x14) [19225.764465] [<c010a0d9>] (show_stack) from [<c0867e29>] (dump_stack+0x69/0x78) [19225.764475] [<c0867e29>] (dump_stack) from [<c086bbef>] (nmi_cpu_backtrace+0x8b/0xd4) [19225.764486] [<c086bbef>] (nmi_cpu_backtrace) from [<c010ca01>] (handle_IPI+0x75/0x278) [19225.764495] [<c010ca01>] (handle_IPI) from [<c01013e3>] (gic_handle_irq+0x67/0x68) [19225.764502] [<c01013e3>] (gic_handle_irq) from [<c010aa25>] (__irq_svc+0x65/0x94) [19225.764506] Exception stack(0xee523e98 to 0xee523ee0) [19225.764510] 3e80: c0df6740 c0df6740 [19225.764519] 3ea0: 00000000 00000080 ffffe000 00000000 00000282 ffffe000 00000001 ee434000 [19225.764527] 3ec0: c0d02080 4000001f 7fffffff ee523ee8 c011ef33 c0101462 40000133 ffffffff [19225.764535] [<c010aa25>] (__irq_svc) from [<c0101462>] (__do_softirq+0x7a/0x25c) [19225.764545] [<c0101462>] (__do_softirq) from [<c011ef33>] (irq_exit+0x7f/0xc4) [19225.764557] [<c011ef33>] (irq_exit) from [<c0159e5d>] (__handle_domain_irq+0x49/0x84) [19225.764567] [<c0159e5d>] (__handle_domain_irq) from [<c01013b5>] (gic_handle_irq+0x39/0x68) [19225.764575] [<c01013b5>] (gic_handle_irq) from [<c010aa25>] (__irq_svc+0x65/0x94) [19225.764578] Exception stack(0xee523f78 to 0xee523fc0) [19225.764582] 3f60: 00000001 00000000 [19225.764590] 3f80: 00000000 c0116561 ffffe000 c0d03fcc c0d03f6c c0cb6438 c0ddd8eb 00000000 [19225.764603] 3fa0: 00000000 00000000 00087b5d ee523fc8 c01070e7 c01070e8 40000033 ffffffff [19225.764613] [<c010aa25>] (__irq_svc) from [<c01070e8>] (arch_cpu_idle+0x28/0x2c) [19225.764624] [<c01070e8>] (arch_cpu_idle) from [<c014c93d>] (do_idle+0x115/0x16c) [19225.764642] [<c014c93d>] (do_idle) from [<c014cb89>] (cpu_startup_entry+0x19/0x1c) [19225.764656] [<c014cb89>] (cpu_startup_entry) from [<401016f1>] (0x401016f1)

So if I understand this correctly ... even though I have the mali.ko, there is no easy way of using it in X currently? How does this differ from using the legacy kernel / stack? I thought the difference was only in the kernel and user-space stuff should be more or less the same. Could you please elaborate? Also, it seems that I only get mali.ko, not mali_drm.ko ... what's the relation and is that what's preventing the use of something useful? I am very inexperienced with the graphics stack, so if those are super silly questions, I am sorry to bother.