OdyX

@ebin-dev I think this is caused by https://github.com/armbian/build/commit/1f4df4c41fe33f9822ca2f42d14a2a445e27aed7 ; Rockchip64's 'edge' kernels were bumped to 6.7.

From the code history, it looks like rk3399 is the "old" family, Helios64 should be BOARDFAMILY=rockchip64 (see https://github.com/armbian/build/blob/main/config/boards/helios64.csc#L3), so it's the code in /usr/lib/armbian/armbian-hardware-optimization that's outdated. I've therefore proposed https://github.com/armbian/build/pull/6159 . Please test and comment there if you can!

Would it make sense to let the kernel pick where it does what? That seems like a weird optimization to have. (I'm using ext4 on cryptsetup on raid5, on 5 spinny 3,5" disks)

No, just some 3-4 docker images (jackett, radarr, sonarr), 2 tor clients, smb, transmission.

I managed to get one of my helios64 crash with the above code indeed, with linux-u-boot-edge-helios64_22.02.1_arm64.deb on kernel 5.15.93 indeed. Armbian 23.8.1 bullseye ttyS2 [ 115.729058] Internal error: Oops: 86000005 [#1] PREEMPT SMP [ 115.729568] Modules linked in: bluetooth unix_diag veth nft_masq nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bridge dm_mod ipt_REJECT nf_reject_ipv4 xt_multiport nft_compat nft_counter nf_tables nfnetlink binfmt_misc rfkill lz4hc lz4 zram raid456 async_memcpy async_raid6_recov async_pq async_xor async_tx md_mod r8152 cdc_acm snd_soc_hdmi_codec snd_soc_rockchip_i2s snd_soc_rockchip_pcm leds_pwm pwm_fan snd_soc_core gpio_charger panfrost snd_pcm_dmaengine snd_pcm gpu_sched snd_timer snd soundcore realtek rockchip_vdec(C) hantro_vpu(C) rockchip_iep rockchip_rga v4l2_h264 videobuf2_dma_contig videobuf2_vmalloc videobuf2_dma_sg v4l2_mem2mem videobuf2_memops fusb302 sg videobuf2_v4l2 videobuf2_common dwmac_rk tcpm stmmac_platform typec videodev mc stmmac pcs_xpcs adc_keys gpio_beeper cpufreq_dt ledtrig_netdev lm75 sunrpc ip_tables x_tables autofs4 [ 115.736491] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G C 5.15.93-rockchip64 #23.02.2 [ 115.737279] Hardware name: Helios64 (DT) [ 115.737631] pstate: 200000c5 (nzCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 115.738252] pc : 0xffff8004080d5e8c [ 115.738573] lr : 0xffff8004080d5e8c [ 115.738887] sp : ffff800009df3e60 [ 115.739185] x29: ffff800009df3e60 x28: ffff00000078bb00 x27: 0000000000000000 [ 115.739826] x26: ffff800009eebc80 x25: 0000000000000001 x24: ffff000000404300 [ 115.740467] x23: 00000000000000c0 x22: ffffffffffffffd0 x21: ffff8000095504a8 [ 115.741105] x20: ffff0000f77ab980 x19: ffffffffffffffd0 x18: 0000000000000000 [ 115.741744] x17: ffff8000ee06c000 x16: ffff800009df4000 x15: 00001f1e8e1e9e92 [ 115.742384] x14: 00000000000003f6 x13: 0000000000000056 x12: 0000000000000000 [ 115.743023] x11: 0000000000000001 x10: 0000000000000000 x9 : 0000000000000056 [ 115.743662] x8 : ffff0000f77aba00 x7 : ffff0000f77aba30 x6 : 0000000000000001 [ 115.744301] x5 : ffff8000ee06c000 x4 : 0000000000010002 x3 : 000000000001b663 [ 115.744940] x2 : ffffffffffffa88d x1 : 00000000ffff4b2f x0 : 000000000000d5ba [ 115.745580] Call trace: [ 115.745802] 0xffff8004080d5e8c [ 115.746088] flush_smp_call_function_queue+0x114/0x250 [ 115.746557] generic_smp_call_function_single_interrupt+0x14/0x20 [ 115.747103] ipi_handler+0x7c/0x340 [ 115.747423] handle_percpu_devid_irq+0xa0/0x240 [ 115.747830] handle_domain_irq+0x90/0xd8 [ 115.748187] gic_handle_irq+0xb8/0x134 [ 115.748528] call_on_irq_stack+0x28/0x50 [ 115.748883] do_interrupt_handler+0x58/0x68 [ 115.749261] el1_interrupt+0x30/0x78 [ 115.749585] el1h_64_irq_handler+0x18/0x28 [ 115.749954] el1h_64_irq+0x74/0x78 [ 115.750261] arch_cpu_idle+0x18/0x28 [ 115.750584] default_idle_call+0x40/0x184 [ 115.750949] do_idle+0x1fc/0x270 [ 115.751245] cpu_startup_entry+0x28/0x50 [ 115.751602] secondary_start_kernel+0x164/0x178 [ 115.752011] __secondary_switched+0x90/0x94 [ 115.752396] Code: bad PC value [ 115.752677] ---[ end trace 0ceb9c6e6a618ff5 ]--- [ 115.753092] Kernel panic - not syncing: Oops: Fatal exception in interrupt [ 115.753699] SMP: stopping secondary CPUs [ 116.920717] SMP: failed to stop secondary CPUs 0,5 [ 116.921146] Kernel Offset: disabled [ 116.921458] CPU features: 0x800820f1,20000846 [ 116.921847] Memory Limit: none [ 116.922129] ---[ end Kernel panic - not syncing: Oops: Fatal exception in interrupt ]---

6.1.71 only got 17 hours of uptime, then 5, so I now tried 6.6, which died after barely 1 hour. Will revert to 5.15.93.

Yeah, I downloaded the dtb from your Dropbox link; thanks for that. Am now rebooting into 6.1.71, and will report back.

Well. With all setup as you said but with the 5.15.93 kernel, it now has a 3-days uptime. Yay.

The only somewhat relevant thing I updated was to use `linux-cpupower` instead of `cpufrequtils`, and I had started using the "ondemand" governor for the CPUs. I see that sbc-bench set the governor for all CPUs at "performance". I've now returned to leaving the settings as set by sbc-bench, to see if my suspicion (aroused by https://github.com/ThomasKaiser/sbc-bench/issues/62) is warranted.

(muffled rage sounds) It worked fine as long as it wasn't doing anything. Now I've added a munin-node, tor, various mounts, and 🎉 it has started to randomly stop (die, kernel panic, whatever, I can't determine what the issue is…). 😕 Any idea how to debug this? /var/log is a zram device (and journal has never shown anything useful). Do I have to resort to using a serial-console permanently to debug this?

That was it; it rebooted without issues with this change, thanks so much @ebin-dev!

Thanks @ebin-dev for these instructions; after I bricked a nicely-working (but constantly shutting down) bullseye Helios64 by "apt upgrade"ing it to bookwork, I went with a (backup and a) clean reinstall following your instructions, and it seems much stabler indeed! (Of course, it's not doing anything yet as I still need to resetup everything I had...). The only issue I have is that it doesn't boot if I have my DELOCK 4-SATA-to-USB-C plugged in (to a USB-3.0 port). Unplugging it, booting and plugging it later works though. Any idea?