Trillien

Hey @Adrienne, Github thread talks about two to three ways to so solve the issue. Did you apply the one that removes the symlinks ?

@BipBip1981, can you please share again your feedback on fancontrol running on Trixie ? The site can't display the image.

Hi, For info I had the same issue running OMV 6.0 with PhotoPrism. It seemed PhotoPrism consumes a lot of CPU to compute the pictures. And at a point (between a quarter and two hours), Helios64 failed with rcu_preempt detected stalls on CPUs error. This error was probably linked to the jump between frequencies : I solved it by limiting the max frequency on biggest cores at 1200 MHz. Note that was before Prahal's DTB to increase cpu voltage.

I've just tested cpufreq-switching-2-delay5msec in emergency mode. My setup is Armbian 23.08.0-trunk Bookworm with Linux 6.6.8-edge-rockchip64 My first attempt ran for few minutes before freezing and rebooting. Before my second attempt, I blacklist panfrost (not sure there is an impact at all in emergency mode...) $ sudo nano /etc/modprobe.d/blacklist.conf blacklist panfrost I then rebooted and started linux in emergency mode. # lsmod | grep panfrost lsmod didn't catch any panfrost module. Still, cpufreq-switching-2-delay5msec ended with a kernel panic: 10/100 cpub_freq 600000 cpub_freq 816000 cpub_freq 1008000 cpub_freq 1200000 cpub_freq 1416000 cpub_freq 1608000 cpub_freq 1800000 cpub_freq 1608000 cpub_freq 1416000 cpub_freq 1200000 cpub_freq 1008000 cpub_freq 816000 cpub_freq 600000 cpub_freq 408000 cpub_freq 600000 cpub_freq 816000 cpub_freq 1008000 cpub_freq 1200000 cpub_freq 1416000 cpub_freq 1608000 cpub_freq 1800000 cpub_freq 1608000 cpub_freq 1416000 cpub_freq 1200000 cpub_freq 1008000 cpub_freq 816000 cpub_freq 600000 cpub_freq 408000 cpub_freq 600000 cpub_freq 816000 cpub_freq 1008000 cpub_freq 1200000 cpub_freq 1416000 cpub_freq 1608000 cpub_freq 1800000 cpub_freq 1608000 cpub_freq 1416000 cpub_freq 1200000 cpub_freq 1008000 cpub_freq 816000 cpub_freq 600000 cpub_freq 408000 cpub_freq 600000 cpub_freq 816000 cpub_freq 1008000 cpub_freq 1200000 cpub_freq 1416000 cpub_freq 1608000 cpub_freq 1800000 cpub_freq 1608000 cpub_freq 1416000 cpub_freq 1200000 cpub_freq 1008000 cpub_freq 816000 cpub_freq 600000 cpub_freq 408000 cpub_freq 600000 cpub_freq 816000 cpub_freq 1008000 cpub_freq 1200000 cpub_freq 1416000 cpub_freq 1608000 cpub_freq 1800000 cpub_freq 1608000 cpub_freq 600000 cpub_freq 816000 cpub_freq 1008000 cpub_freq 1200000 cpub_freq 1416000 cpub_freq 1608000 cpub_freq 1800000 cpub_freq 1608000 cpub_freq 1416000 cpub_freq 1200000 cpub_freq 1008000 cpub_freq 816000 cpub_freq 600000 cpub_freq 408000 cpub_freq 600000 cpub_freq 816000 cpub_freq 1008000 cpub_freq 1200000 cpub_freq 1416000 cpub_freq 1608000 cpub_freq 1800000 cpub_freq 1608000 cpub_freq 1416000 cpub_freq 1200000 cpub_freq 1008000 cpub_freq 816000 cpub_freq 600000 cpub_freq 408000 cpub_freq 600000 cpub_freq 816000 cpub_freq 1008000 [ 51.732314] Internal error: Oops: 0000000096000006 [#1] PREEMPT SMP [ 51.732887] Modules linked in: ip_tables x_tables autofs4 efivarfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx raid0 multipath linear cdc_ncm cdc_ether usbnet raid1 r8152 realtek rockchipdrm dw_mipi_dsi dw_hdmi analogix_dp fusb302 drm_display_helper cec tcpm drm_dma_helper typec drm_kms_helper dwmac_rk stmmac_platform stmmac drm pcs_xpcs adc_keys [ 51.735962] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 6.6.8-edge-rockchip64 #1 [ 51.736610] Hardware name: Helios64 (DT) [ 51.736965] pstate: 800000c5 (Nzcv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 51.737589] pc : update_blocked_averages+0x278/0x758 [ 51.738051] lr : update_blocked_averages+0x264/0x758 [ 51.738504] sp : ffff800082043e80 [ 51.738806] x29: ffff800082043e80 x28: ffff000005cd9600 x27: 0000000c01b0a9aa [ 51.739456] x26: ffff000005cdbc80 x25: ffff000005cdbc00 x24: 0000000000000000 [ 51.740104] x23: ffff0000f77a0f18 x22: 0000000000000028 x21: 0000000000000000 [ 51.740750] x20: ffff000005cdbd40 x19: ffff0000f77a05c0 x18: 0000000000000000 [ 51.741396] x17: ffff800075e85000 x16: ffff800082040000 x15: 0000000000000000 [ 51.742043] x14: 0000000000000001 x13: 000000000000002a x12: 00000000000e7fe0 [ 51.742689] x11: 0000000000000000 x10: 000000000000002a x9 : ffff000005cdbc80 [ 51.743336] x8 : 0000000000000000 x7 : ffff000005cdbc00 x6 : 0000000000000014 [ 51.743982] x5 : 00000000000003af x4 : 000000000000b6a8 x3 : 0000000000000000 [ 51.744628] x2 : 0000000000000000 x1 : ffff000001d6c400 x0 : ffff0000019c6000 [ 51.745274] Call trace: [ 51.745501] update_blocked_averages+0x278/0x758 [ 51.745924] run_rebalance_domains+0x4c/0x80 [ 51.746313] __do_softirq+0x160/0x3fc [ 51.746649] ____do_softirq+0x10/0x1c [ 51.746985] call_on_irq_stack+0x24/0x4c [ 51.747344] do_softirq_own_stack+0x1c/0x2c [ 51.747727] irq_exit_rcu+0x94/0xd0 [ 51.748051] el1_interrupt+0x38/0x68 [ 51.748382] el1h_64_irq_handler+0x18/0x24 [ 51.748758] el1h_64_irq+0x64/0x68 [ 51.749070] cpuidle_enter_state+0xc0/0x4bc [ 51.749454] cpuidle_enter+0x38/0x50 [ 51.749783] do_idle+0x1fc/0x270 [ 51.750083] cpu_startup_entry+0x34/0x3c [ 51.750444] secondary_start_kernel+0x128/0x148 [ 51.750859] __secondary_switched+0xb8/0xbc [ 51.751249] Code: f940ab20 f9406400 f8766801 b4000101 (f9407020) [ 51.751797] ---[ end trace 0000000000000000 ]--- [ 51.752214] Kernel panic - not syncing: Oops: Fatal exception in interrupt [ 51.752826] SMP: stopping secondary CPUs [ 51.753358] Kernel Offset: disabled [ 51.753674] CPU features: 0x1,00000208,3c020000,1000421b [ 51.754152] Memory Limit: none [ 51.754439] ---[ end Kernel panic - not syncing: Oops: Fatal exception in interrupt ]---

Hi @prahal, Yes, for few weeks I took the habit to use the serial console to better understand whether the boot crashes or is stuck at some steps (I've got a long long network config step with systemd-networkd). The message appears after u-boot gives the hand to linux, and is part of the very first lines where linux checks the hard drive filesystem.

Hi @TDCroPower Have you got any issue with systemd-networkd service ? I systematically have a timeout as it seems waiting for all network interface to be ready. $ systemctl status systemd-networkd-wait-online.service × systemd-networkd-wait-online.service - Wait for Network to be Configured Loaded: loaded (/lib/systemd/system/systemd-networkd-wait-online.service; enabled; preset: disabled) Drop-In: /etc/systemd/system/systemd-networkd-wait-online.service.d └─override.conf Active: failed (Result: exit-code) since Fri 2024-04-19 16:33:57 CEST; 18min ago Docs: man:systemd-networkd-wait-online.service(8) Process: 1188 ExecStart=/lib/systemd/systemd-networkd-wait-online (code=exited, status=1/FAILURE) Main PID: 1188 (code=exited, status=1/FAILURE) CPU: 48ms I use end0 network interface. The other enx646266d00b79 isn't connected I've got a bridge br0 with a virtual tap0 interface (OpenVPN) podman runs a container and enable the interface veth7198de08 and the bridge cni-podman. $ networkctl IDX LINK TYPE OPERATIONAL SETUP 1 lo loopback carrier unmanaged 2 end0 ether enslaved configuring 4 enx646266d00b79 ether no-carrier configured 5 br0 bridge routable configured 6 tap0 ether enslaved configuring 7 cni-podman0 bridge routable unmanaged 8 veth7198de08 ether enslaved unmanaged 7 links listed.

By the way, Running Armbian 23.08.0-trunk Bookworm with Linux 6.6.8-edge-rockchip64 I notice an error at the beginning of linux boot. "mdadm: initramfs boot message: /scripts/local-bottom/mdadm: rm: not found" Seems an old one as I follow these instructions for Helios4 to solve the issue https://wiki.kobol.io/helios4/mdadm/#fix-mdadm

Hi @prahal I've just done a test with your cpufreq-switching-2 program. I'm running Helios64 on Armbian 23.08.0-trunk Bookworm with Linux 6.6.8-edge-rockchip64 I've started with LITTLE (CPUL = 1) The program ran the 100 loops without issue. Then I ran with big (CPUB = 1) So far it failed at the 6th loop Before a third run, I tried to change the interrupt allocation on xhci and ahci as you suggested Please note the interrupts may vary after reboot (e.g. ahci was 76-80, after reboot it is 75-79) # cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 18: 0 0 0 0 0 0 GICv3 25 Level vgic 20: 0 0 0 0 0 0 GICv3 27 Level kvm guest vtimer 23: 7947 8876 6014 7156 18916 24271 GICv3 30 Level arch_timer 25: 6601 5232 4476 4609 11249 4343 GICv3 113 Level rk_timer 31: 0 0 0 0 0 0 GICv3 37 Level ff6d0000.dma-controller 32: 0 0 0 0 0 0 GICv3 38 Level ff6d0000.dma-controller 33: 0 0 0 0 0 0 GICv3 39 Level ff6e0000.dma-controller 34: 0 0 0 0 0 0 GICv3 40 Level ff6e0000.dma-controller 36: 915 0 0 0 0 0 GICv3 132 Level ttyS2 37: 0 0 0 0 0 0 GICv3 147 Level ff650800.iommu 38: 0 0 0 0 0 0 GICv3 149 Level ff660480.iommu 39: 0 0 0 0 0 0 GICv3 151 Level ff8f3f00.iommu, ff8f0000.vop 40: 0 0 0 0 0 0 GICv3 150 Level ff903f00.iommu, ff900000.vop 41: 0 0 0 0 0 0 GICv3 75 Level ff914000.iommu 42: 0 0 0 0 0 0 GICv3 76 Level ff924000.iommu 43: 0 0 0 0 0 0 GICv3 85 Level ff1d0000.spi 44: 0 0 0 0 0 0 GICv3 84 Level ff1e0000.spi 45: 0 0 0 0 0 0 GICv3 164 Level ff200000.spi 46: 1399 0 0 0 1775 0 GICv3 142 Level xhci-hcd:usb1 47: 30 0 0 0 0 0 GICv3 67 Level ff120000.i2c 48: 0 0 0 0 0 0 GICv3 68 Level ff160000.i2c 49: 5031 0 0 0 0 0 GICv3 89 Level ff3c0000.i2c 50: 540 0 0 0 0 0 GICv3 88 Level ff3d0000.i2c 51: 0 0 0 0 0 0 GICv3 90 Level ff3e0000.i2c 52: 0 0 0 0 0 0 GICv3 129 Level rockchip_thermal 53: 0 0 0 0 0 0 GICv3 152 Edge ff848000.watchdog 54: 0 0 0 0 0 0 GICv3-23 0 Level arm-pmu 55: 0 0 0 0 0 0 GICv3-23 1 Level arm-pmu 56: 0 0 0 0 0 0 rockchip_gpio_irq 9 Edge 2-0020 57: 0 0 0 0 0 0 rockchip_gpio_irq 10 Level rk808 63: 0 0 0 0 0 0 rk808 5 Edge RTC alarm 67: 2 0 0 0 0 0 GICv3 94 Level ff100000.saradc 68: 0 0 0 0 0 0 GICv3 97 Level dw-mci 69: 0 0 0 0 0 0 rockchip_gpio_irq 7 Edge fe320000.mmc cd 70: 0 0 0 0 0 0 GICv3 81 Level pcie-sys 72: 0 0 0 0 0 0 GICv3 83 Level pcie-client 74: 0 0 0 0 0 0 ITS-MSI 0 Edge PCIe PME, aerdrv 75: 0 489 0 0 524 0 ITS-MSI 524288 Edge ahci0 76: 0 0 237 0 0 904 ITS-MSI 524289 Edge ahci1 77: 0 0 0 489 31578 0 ITS-MSI 524290 Edge ahci2 78: 0 0 0 0 249 0 ITS-MSI 524291 Edge ahci3 79: 0 0 0 0 0 248 ITS-MSI 524292 Edge ahci4 83: 14093 0 0 0 0 0 GICv3 43 Level mmc1 84: 0 0 0 0 0 0 rockchip_gpio_irq 5 Edge Power 85: 0 0 0 0 0 0 rockchip_gpio_irq 3 Edge User Button 1 86: 0 0 0 931 0 0 GICv3 44 Level end0 87: 5 0 0 0 0 0 rockchip_gpio_irq 2 Level fsc_interrupt_int_n 88: 0 0 0 0 0 0 GICv3 59 Level rockchip_usb2phy 89: 0 0 0 0 0 0 GICv3 135 Level rockchip_usb2phy_bvalid 90: 0 0 0 0 0 0 GICv3 136 Level rockchip_usb2phy_id 91: 0 0 0 0 0 0 GICv3 60 Level ohci_hcd:usb4 92: 0 0 0 0 0 0 GICv3 58 Level ehci_hcd:usb3 93: 0 0 0 0 0 0 GICv3 137 Level dwc3-otg, xhci-hcd:usb5 94: 0 0 0 0 0 0 GICv3 32 Level rk-crypto 95: 0 0 0 0 0 0 GICv3 146 Level ff650000.video-codec 96: 0 0 0 0 0 0 GICv3 87 Level ff680000.rga 97: 0 0 0 0 0 0 GICv3 145 Level ff650000.video-codec 98: 0 0 0 0 0 0 GICv3 148 Level ff660000.video-codec 99: 0 0 0 0 0 0 rockchip_gpio_irq 2 Edge gpio-charger 100: 0 0 0 0 0 0 rockchip_gpio_irq 27 Edge gpio-charger 101: 2 0 0 0 0 0 GICv3 51 Level panfrost-gpu 102: 0 0 0 0 0 0 GICv3 53 Level panfrost-mmu 103: 0 0 0 0 0 0 GICv3 52 Level panfrost-job IPI0: 1384 1517 1472 1311 4816 7551 Rescheduling interrupts IPI1: 12225 10971 9100 9240 10161 26978 Function call interrupts IPI2: 0 0 0 0 0 0 CPU stop interrupts IPI3: 0 0 0 0 0 0 CPU stop (for crash dump) interrupts IPI4: 2213 2003 2357 2402 2137 1671 Timer broadcast interrupts IPI5: 598 601 747 496 1106 784 IRQ work interrupts IPI6: 0 0 0 0 0 0 CPU wake-up interrupts Err: 0 I reallocated the interrupts over the little core. # echo 0 > /proc/irq/46/smp_affinity_list # echo 1 > /proc/irq/75/smp_affinity_list # echo 2 > /proc/irq/76/smp_affinity_list # echo 3 > /proc/irq/77/smp_affinity_list # echo 0 > /proc/irq/78/smp_affinity_list # echo 1 > /proc/irq/79/smp_affinity_list Then I ran the program on the big again (CPUB = 1) And I reach the 25th loop before it failed.

I was curious about how cpufrequtils can manage different policies over several cpus. Actually it can only dictate one set over the 6 cpus. For info, I've just found a post about having a change on cpufrequtils that can manage different frequency sets over the cpus (4x cpus at 400-1400MHz and 2x cpus at 400-1800MHz). #To get individual cpus current frequencies $ cat /sys/devices/system/cpu/cpu*/cpufreq/cpuinfo_cur_freq #To get individual cpus governor (note there is no one governor for one cpu) $ cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor #To collect existing frequency policies $ ls -al /sys/devices/system/cpu/cpufreq/ I'm now testing the script... Edit: Not probant ! It reboots with "400-1400MHz schedutil" and "400-1800MHz conservative" governors.

Here is a short feedback on my Helios64 experience. Few years ago I had troubles at having a stable system running OMV5 on Buster. I decided to move to Bullseye and OM6 and finally achieved a stable environment. Absolutely no reboot, but for OMV6 Photoprism extension I couldn't run. I recently tried to move to a fresh Bookworm install with OMV7. I felt it was painful to track within the forum each and every necessary changes to the image. It took me 2 tries before I apprehended the effect of the changes, and I carefully followed the instructions to apply the changes in the right order. Since my last install 5 days ago, it seems I now have something stable again.

For info, probably related to this issue : https://github.com/openwrt/openwrt/issues/14377

Nop, I didn't get a chance to reach these steps at the first install. It reboot during update-initramfs, part of apt install openmediavault substep. However, /etc/default/cpufrequtils were already changed to "schedutil"

I plan to use OMV Photoprism extension that's managed as a container by OMV. As I remember, OMV doesn't manage containers with Docker but Podman. Not tested yet. Did you have a look at?

Hi @prahal, I'm running Armbian_23.5.4_Helios64_bookworm_current_6.1.36 on my Helios64. In /etc/armbian-release, BOARDFAMILY was rk3399 As per your post, I manually changed BOARDFAMILY to rockchip64. $ cat /etc/armbian-release # PLEASE DO NOT EDIT THIS FILE BOARD=helios64 BOARD_NAME="Helios64" BOARDFAMILY=rockchip64 BUILD_REPOSITORY_URL=https://github.com/armbian/build BUILD_REPOSITORY_COMMIT=173ed85 VERSION=23.08.0-trunk LINUXFAMILY=rockchip64 ARCH=arm64 IMAGE_TYPE=stable BOARD_TYPE=csc INITRD_ARCH=arm64 KERNEL_IMAGE_TYPE=Image FORCE_BOOTSCRIPT_UPDATE= VENDOR=Armbian REVISION=23.08.0-trunk BOOTSCRIPT_FORCE_UPDATE="no" BOOTSCRIPT_DST="boot.cmd" BRANCH=current That's dirty, but that let me have access to the newest armbian package releases. Can you please advise whether you know a cleaner way to match to the new BOARDFAMILY for Helios64 and its rockchip processor?

Sorry, I meant dpkg --configure -a I've got a theory on /etc/default/cpufrequtils change. Searching within openmediavault github repository for "Deploying Service Configurations ...", there is only one result : deb/openmediavault/debian/openmediavault.postinst According to openmediavault.postinst, the script echoes this message and runs omv-salt deploy run cpufrequtils 237 ######################################################################## 238 # Deploy various services when the package is installed the first time. 239 ######################################################################## 240 if [ -z "$2" ]; then 241 # Disable default nginx site. 242 nginx_dissite default || : 243 244 # Deploy the configuration for various services. 245 echo "Deploying service configurations ..." 246 LANG=C.UTF-8 omv-salt deploy run --no-color --quiet \ 247 apt cpufrequtils chrony rsyslog \ 248 watchdog monit rrdcached avahi ssh nginx collectd \ 249 phpfpm issue sysctl systemd systemd-logind || : 250 fi Thanks to https://docs.openmediavault.org/en/stable/various/advset.html#environmental-variables, I learn the configuration files used by omv-salt is stored at /srv/salt/omv Looking at the deploy stage in /srv/salt/omv I find a cpufrequtils folder kobol@helios64:~$ ls -al /srv/salt/omv/deploy/cpufrequtils/ total 20 drwxr-xr-x 3 root root 4096 Apr 5 15:58 . drwxr-xr-x 39 root root 4096 Apr 5 17:04 .. -rw-r--r-- 1 root root 2387 Mar 9 12:58 default.sls drwxr-xr-x 2 root root 4096 Apr 5 15:58 files -rw-r--r-- 1 root root 916 Mar 9 12:58 init.sls default.sls tells us the source file used by salt to configure /etc/default/cpufrequtils is files/cpufrequtils.j2 $ nano /srv/salt/omv/deploy/cpufrequtils/default.sls ... configure_default_cpufrequtils: file.managed: - name: "/etc/default/cpufrequtils" - source: - salt://{{ tpldir }}/files/cpufrequtils.j2 - template: jinja - context: cpufreq: {{ config.cpufreq }} - user: root - group: root - mode: 644 ... Finally, files/cpufrequtils.j2 calls for OMV_CPUFREQUTILS_MAXSPEED, OMV_CPUFREQUTILS_MINSPEED and OMV_CPUFREQUTILS_GOVERNOR $ nano /srv/salt/omv/deploy/cpufrequtils/files/cpufrequtils.j2 {%- set max_speed = salt['pillar.get']('default:OMV_CPUFREQUTILS_MAXSPEED', '0') -%} {%- set min_speed = salt['pillar.get']('default:OMV_CPUFREQUTILS_MINSPEED', '0') -%} {%- set governor = salt['pillar.get']('default:OMV_CPUFREQUTILS_GOVERNOR', salt['grains.filter_by']({ "default": "ondemand", "amd64": "conservative", "i386": "conservative", "armhf": "schedutil", "arm64": "schedutil", "armel": "schedutil" }, grain="osarch")) -%} {%- if cpufreq -%} ENABLE="true" {%- else -%} ENABLE="false" {%- endif %} GOVERNOR="{{ governor }}" MAX_SPEED="{{ max_speed }}" MIN_SPEED="{{ min_speed }}" The point is these values are first defined in the install script at line 543... after omv-salt is called on cpufrequtils at line 337 543 # set defaults in /etc/default/openmediavault 544 omv_set_default "OMV_CPUFREQUTILS_GOVERNOR" "${GOVERNOR}" 545 omv_set_default "OMV_CPUFREQUTILS_MINSPEED" "${MIN_SPEED}" 546 omv_set_default "OMV_CPUFREQUTILS_MAXSPEED" "${MAX_SPEED}" As per /srv/salt/omv/deploy/cpufrequtils/files/cpufrequtils.j2, there isn't any values defined yet in /etc/default/openmediavault. Thus : governor takes "schedutil" max_speed is 0 min_speed is 0