Trillien

Hi, For info I had the same issue running OMV 6.0 with PhotoPrism. It seemed PhotoPrism consumes a lot of CPU to compute the pictures. And at a point (between a quarter and two hours), Helios64 failed with rcu_preempt detected stalls on CPUs error. This error was probably linked to the jump between frequencies : I solved it by limiting the max frequency on biggest cores at 1200 MHz. Note that was before Prahal's DTB to increase cpu voltage.

I've just tested cpufreq-switching-2-delay5msec in emergency mode. My setup is Armbian 23.08.0-trunk Bookworm with Linux 6.6.8-edge-rockchip64 My first attempt ran for few minutes before freezing and rebooting. Before my second attempt, I blacklist panfrost (not sure there is an impact at all in emergency mode...) $ sudo nano /etc/modprobe.d/blacklist.conf blacklist panfrost I then rebooted and started linux in emergency mode. # lsmod | grep panfrost lsmod didn't catch any panfrost module. Still, cpufreq-switching-2-delay5msec ended with a kernel panic: 10/100 cpub_freq 600000 cpub_freq 816000 cpub_freq 1008000 cpub_freq 1200000 cpub_freq 1416000 cpub_freq 1608000 cpub_freq 1800000 cpub_freq 1608000 cpub_freq 1416000 cpub_freq 1200000 cpub_freq 1008000 cpub_freq 816000 cpub_freq 600000 cpub_freq 408000 cpub_freq 600000 cpub_freq 816000 cpub_freq 1008000 cpub_freq 1200000 cpub_freq 1416000 cpub_freq 1608000 cpub_freq 1800000 cpub_freq 1608000 cpub_freq 1416000 cpub_freq 1200000 cpub_freq 1008000 cpub_freq 816000 cpub_freq 600000 cpub_freq 408000 cpub_freq 600000 cpub_freq 816000 cpub_freq 1008000 cpub_freq 1200000 cpub_freq 1416000 cpub_freq 1608000 cpub_freq 1800000 cpub_freq 1608000 cpub_freq 1416000 cpub_freq 1200000 cpub_freq 1008000 cpub_freq 816000 cpub_freq 600000 cpub_freq 408000 cpub_freq 600000 cpub_freq 816000 cpub_freq 1008000 cpub_freq 1200000 cpub_freq 1416000 cpub_freq 1608000 cpub_freq 1800000 cpub_freq 1608000 cpub_freq 1416000 cpub_freq 1200000 cpub_freq 1008000 cpub_freq 816000 cpub_freq 600000 cpub_freq 408000 cpub_freq 600000 cpub_freq 816000 cpub_freq 1008000 cpub_freq 1200000 cpub_freq 1416000 cpub_freq 1608000 cpub_freq 1800000 cpub_freq 1608000 cpub_freq 600000 cpub_freq 816000 cpub_freq 1008000 cpub_freq 1200000 cpub_freq 1416000 cpub_freq 1608000 cpub_freq 1800000 cpub_freq 1608000 cpub_freq 1416000 cpub_freq 1200000 cpub_freq 1008000 cpub_freq 816000 cpub_freq 600000 cpub_freq 408000 cpub_freq 600000 cpub_freq 816000 cpub_freq 1008000 cpub_freq 1200000 cpub_freq 1416000 cpub_freq 1608000 cpub_freq 1800000 cpub_freq 1608000 cpub_freq 1416000 cpub_freq 1200000 cpub_freq 1008000 cpub_freq 816000 cpub_freq 600000 cpub_freq 408000 cpub_freq 600000 cpub_freq 816000 cpub_freq 1008000 [ 51.732314] Internal error: Oops: 0000000096000006 [#1] PREEMPT SMP [ 51.732887] Modules linked in: ip_tables x_tables autofs4 efivarfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx raid0 multipath linear cdc_ncm cdc_ether usbnet raid1 r8152 realtek rockchipdrm dw_mipi_dsi dw_hdmi analogix_dp fusb302 drm_display_helper cec tcpm drm_dma_helper typec drm_kms_helper dwmac_rk stmmac_platform stmmac drm pcs_xpcs adc_keys [ 51.735962] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 6.6.8-edge-rockchip64 #1 [ 51.736610] Hardware name: Helios64 (DT) [ 51.736965] pstate: 800000c5 (Nzcv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 51.737589] pc : update_blocked_averages+0x278/0x758 [ 51.738051] lr : update_blocked_averages+0x264/0x758 [ 51.738504] sp : ffff800082043e80 [ 51.738806] x29: ffff800082043e80 x28: ffff000005cd9600 x27: 0000000c01b0a9aa [ 51.739456] x26: ffff000005cdbc80 x25: ffff000005cdbc00 x24: 0000000000000000 [ 51.740104] x23: ffff0000f77a0f18 x22: 0000000000000028 x21: 0000000000000000 [ 51.740750] x20: ffff000005cdbd40 x19: ffff0000f77a05c0 x18: 0000000000000000 [ 51.741396] x17: ffff800075e85000 x16: ffff800082040000 x15: 0000000000000000 [ 51.742043] x14: 0000000000000001 x13: 000000000000002a x12: 00000000000e7fe0 [ 51.742689] x11: 0000000000000000 x10: 000000000000002a x9 : ffff000005cdbc80 [ 51.743336] x8 : 0000000000000000 x7 : ffff000005cdbc00 x6 : 0000000000000014 [ 51.743982] x5 : 00000000000003af x4 : 000000000000b6a8 x3 : 0000000000000000 [ 51.744628] x2 : 0000000000000000 x1 : ffff000001d6c400 x0 : ffff0000019c6000 [ 51.745274] Call trace: [ 51.745501] update_blocked_averages+0x278/0x758 [ 51.745924] run_rebalance_domains+0x4c/0x80 [ 51.746313] __do_softirq+0x160/0x3fc [ 51.746649] ____do_softirq+0x10/0x1c [ 51.746985] call_on_irq_stack+0x24/0x4c [ 51.747344] do_softirq_own_stack+0x1c/0x2c [ 51.747727] irq_exit_rcu+0x94/0xd0 [ 51.748051] el1_interrupt+0x38/0x68 [ 51.748382] el1h_64_irq_handler+0x18/0x24 [ 51.748758] el1h_64_irq+0x64/0x68 [ 51.749070] cpuidle_enter_state+0xc0/0x4bc [ 51.749454] cpuidle_enter+0x38/0x50 [ 51.749783] do_idle+0x1fc/0x270 [ 51.750083] cpu_startup_entry+0x34/0x3c [ 51.750444] secondary_start_kernel+0x128/0x148 [ 51.750859] __secondary_switched+0xb8/0xbc [ 51.751249] Code: f940ab20 f9406400 f8766801 b4000101 (f9407020) [ 51.751797] ---[ end trace 0000000000000000 ]--- [ 51.752214] Kernel panic - not syncing: Oops: Fatal exception in interrupt [ 51.752826] SMP: stopping secondary CPUs [ 51.753358] Kernel Offset: disabled [ 51.753674] CPU features: 0x1,00000208,3c020000,1000421b [ 51.754152] Memory Limit: none [ 51.754439] ---[ end Kernel panic - not syncing: Oops: Fatal exception in interrupt ]---

Hi @prahal, Yes, for few weeks I took the habit to use the serial console to better understand whether the boot crashes or is stuck at some steps (I've got a long long network config step with systemd-networkd). The message appears after u-boot gives the hand to linux, and is part of the very first lines where linux checks the hard drive filesystem.

Hi @TDCroPower Have you got any issue with systemd-networkd service ? I systematically have a timeout as it seems waiting for all network interface to be ready. $ systemctl status systemd-networkd-wait-online.service × systemd-networkd-wait-online.service - Wait for Network to be Configured Loaded: loaded (/lib/systemd/system/systemd-networkd-wait-online.service; enabled; preset: disabled) Drop-In: /etc/systemd/system/systemd-networkd-wait-online.service.d └─override.conf Active: failed (Result: exit-code) since Fri 2024-04-19 16:33:57 CEST; 18min ago Docs: man:systemd-networkd-wait-online.service(8) Process: 1188 ExecStart=/lib/systemd/systemd-networkd-wait-online (code=exited, status=1/FAILURE) Main PID: 1188 (code=exited, status=1/FAILURE) CPU: 48ms I use end0 network interface. The other enx646266d00b79 isn't connected I've got a bridge br0 with a virtual tap0 interface (OpenVPN) podman runs a container and enable the interface veth7198de08 and the bridge cni-podman. $ networkctl IDX LINK TYPE OPERATIONAL SETUP 1 lo loopback carrier unmanaged 2 end0 ether enslaved configuring 4 enx646266d00b79 ether no-carrier configured 5 br0 bridge routable configured 6 tap0 ether enslaved configuring 7 cni-podman0 bridge routable unmanaged 8 veth7198de08 ether enslaved unmanaged 7 links listed.

By the way, Running Armbian 23.08.0-trunk Bookworm with Linux 6.6.8-edge-rockchip64 I notice an error at the beginning of linux boot. "mdadm: initramfs boot message: /scripts/local-bottom/mdadm: rm: not found" Seems an old one as I follow these instructions for Helios4 to solve the issue https://wiki.kobol.io/helios4/mdadm/#fix-mdadm

Hi @prahal I've just done a test with your cpufreq-switching-2 program. I'm running Helios64 on Armbian 23.08.0-trunk Bookworm with Linux 6.6.8-edge-rockchip64 I've started with LITTLE (CPUL = 1) The program ran the 100 loops without issue. Then I ran with big (CPUB = 1) So far it failed at the 6th loop Before a third run, I tried to change the interrupt allocation on xhci and ahci as you suggested Please note the interrupts may vary after reboot (e.g. ahci was 76-80, after reboot it is 75-79) # cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 18: 0 0 0 0 0 0 GICv3 25 Level vgic 20: 0 0 0 0 0 0 GICv3 27 Level kvm guest vtimer 23: 7947 8876 6014 7156 18916 24271 GICv3 30 Level arch_timer 25: 6601 5232 4476 4609 11249 4343 GICv3 113 Level rk_timer 31: 0 0 0 0 0 0 GICv3 37 Level ff6d0000.dma-controller 32: 0 0 0 0 0 0 GICv3 38 Level ff6d0000.dma-controller 33: 0 0 0 0 0 0 GICv3 39 Level ff6e0000.dma-controller 34: 0 0 0 0 0 0 GICv3 40 Level ff6e0000.dma-controller 36: 915 0 0 0 0 0 GICv3 132 Level ttyS2 37: 0 0 0 0 0 0 GICv3 147 Level ff650800.iommu 38: 0 0 0 0 0 0 GICv3 149 Level ff660480.iommu 39: 0 0 0 0 0 0 GICv3 151 Level ff8f3f00.iommu, ff8f0000.vop 40: 0 0 0 0 0 0 GICv3 150 Level ff903f00.iommu, ff900000.vop 41: 0 0 0 0 0 0 GICv3 75 Level ff914000.iommu 42: 0 0 0 0 0 0 GICv3 76 Level ff924000.iommu 43: 0 0 0 0 0 0 GICv3 85 Level ff1d0000.spi 44: 0 0 0 0 0 0 GICv3 84 Level ff1e0000.spi 45: 0 0 0 0 0 0 GICv3 164 Level ff200000.spi 46: 1399 0 0 0 1775 0 GICv3 142 Level xhci-hcd:usb1 47: 30 0 0 0 0 0 GICv3 67 Level ff120000.i2c 48: 0 0 0 0 0 0 GICv3 68 Level ff160000.i2c 49: 5031 0 0 0 0 0 GICv3 89 Level ff3c0000.i2c 50: 540 0 0 0 0 0 GICv3 88 Level ff3d0000.i2c 51: 0 0 0 0 0 0 GICv3 90 Level ff3e0000.i2c 52: 0 0 0 0 0 0 GICv3 129 Level rockchip_thermal 53: 0 0 0 0 0 0 GICv3 152 Edge ff848000.watchdog 54: 0 0 0 0 0 0 GICv3-23 0 Level arm-pmu 55: 0 0 0 0 0 0 GICv3-23 1 Level arm-pmu 56: 0 0 0 0 0 0 rockchip_gpio_irq 9 Edge 2-0020 57: 0 0 0 0 0 0 rockchip_gpio_irq 10 Level rk808 63: 0 0 0 0 0 0 rk808 5 Edge RTC alarm 67: 2 0 0 0 0 0 GICv3 94 Level ff100000.saradc 68: 0 0 0 0 0 0 GICv3 97 Level dw-mci 69: 0 0 0 0 0 0 rockchip_gpio_irq 7 Edge fe320000.mmc cd 70: 0 0 0 0 0 0 GICv3 81 Level pcie-sys 72: 0 0 0 0 0 0 GICv3 83 Level pcie-client 74: 0 0 0 0 0 0 ITS-MSI 0 Edge PCIe PME, aerdrv 75: 0 489 0 0 524 0 ITS-MSI 524288 Edge ahci0 76: 0 0 237 0 0 904 ITS-MSI 524289 Edge ahci1 77: 0 0 0 489 31578 0 ITS-MSI 524290 Edge ahci2 78: 0 0 0 0 249 0 ITS-MSI 524291 Edge ahci3 79: 0 0 0 0 0 248 ITS-MSI 524292 Edge ahci4 83: 14093 0 0 0 0 0 GICv3 43 Level mmc1 84: 0 0 0 0 0 0 rockchip_gpio_irq 5 Edge Power 85: 0 0 0 0 0 0 rockchip_gpio_irq 3 Edge User Button 1 86: 0 0 0 931 0 0 GICv3 44 Level end0 87: 5 0 0 0 0 0 rockchip_gpio_irq 2 Level fsc_interrupt_int_n 88: 0 0 0 0 0 0 GICv3 59 Level rockchip_usb2phy 89: 0 0 0 0 0 0 GICv3 135 Level rockchip_usb2phy_bvalid 90: 0 0 0 0 0 0 GICv3 136 Level rockchip_usb2phy_id 91: 0 0 0 0 0 0 GICv3 60 Level ohci_hcd:usb4 92: 0 0 0 0 0 0 GICv3 58 Level ehci_hcd:usb3 93: 0 0 0 0 0 0 GICv3 137 Level dwc3-otg, xhci-hcd:usb5 94: 0 0 0 0 0 0 GICv3 32 Level rk-crypto 95: 0 0 0 0 0 0 GICv3 146 Level ff650000.video-codec 96: 0 0 0 0 0 0 GICv3 87 Level ff680000.rga 97: 0 0 0 0 0 0 GICv3 145 Level ff650000.video-codec 98: 0 0 0 0 0 0 GICv3 148 Level ff660000.video-codec 99: 0 0 0 0 0 0 rockchip_gpio_irq 2 Edge gpio-charger 100: 0 0 0 0 0 0 rockchip_gpio_irq 27 Edge gpio-charger 101: 2 0 0 0 0 0 GICv3 51 Level panfrost-gpu 102: 0 0 0 0 0 0 GICv3 53 Level panfrost-mmu 103: 0 0 0 0 0 0 GICv3 52 Level panfrost-job IPI0: 1384 1517 1472 1311 4816 7551 Rescheduling interrupts IPI1: 12225 10971 9100 9240 10161 26978 Function call interrupts IPI2: 0 0 0 0 0 0 CPU stop interrupts IPI3: 0 0 0 0 0 0 CPU stop (for crash dump) interrupts IPI4: 2213 2003 2357 2402 2137 1671 Timer broadcast interrupts IPI5: 598 601 747 496 1106 784 IRQ work interrupts IPI6: 0 0 0 0 0 0 CPU wake-up interrupts Err: 0 I reallocated the interrupts over the little core. # echo 0 > /proc/irq/46/smp_affinity_list # echo 1 > /proc/irq/75/smp_affinity_list # echo 2 > /proc/irq/76/smp_affinity_list # echo 3 > /proc/irq/77/smp_affinity_list # echo 0 > /proc/irq/78/smp_affinity_list # echo 1 > /proc/irq/79/smp_affinity_list Then I ran the program on the big again (CPUB = 1) And I reach the 25th loop before it failed.

I was curious about how cpufrequtils can manage different policies over several cpus. Actually it can only dictate one set over the 6 cpus. For info, I've just found a post about having a change on cpufrequtils that can manage different frequency sets over the cpus (4x cpus at 400-1400MHz and 2x cpus at 400-1800MHz). #To get individual cpus current frequencies $ cat /sys/devices/system/cpu/cpu*/cpufreq/cpuinfo_cur_freq #To get individual cpus governor (note there is no one governor for one cpu) $ cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor #To collect existing frequency policies $ ls -al /sys/devices/system/cpu/cpufreq/ I'm now testing the script... Edit: Not probant ! It reboots with "400-1400MHz schedutil" and "400-1800MHz conservative" governors.

Here is a short feedback on my Helios64 experience. Few years ago I had troubles at having a stable system running OMV5 on Buster. I decided to move to Bullseye and OM6 and finally achieved a stable environment. Absolutely no reboot, but for OMV6 Photoprism extension I couldn't run. I recently tried to move to a fresh Bookworm install with OMV7. I felt it was painful to track within the forum each and every necessary changes to the image. It took me 2 tries before I apprehended the effect of the changes, and I carefully followed the instructions to apply the changes in the right order. Since my last install 5 days ago, it seems I now have something stable again.

For info, probably related to this issue : https://github.com/openwrt/openwrt/issues/14377

Nop, I didn't get a chance to reach these steps at the first install. It reboot during update-initramfs, part of apt install openmediavault substep. However, /etc/default/cpufrequtils were already changed to "schedutil"

I plan to use OMV Photoprism extension that's managed as a container by OMV. As I remember, OMV doesn't manage containers with Docker but Podman. Not tested yet. Did you have a look at?

Hi @prahal, I'm running Armbian_23.5.4_Helios64_bookworm_current_6.1.36 on my Helios64. In /etc/armbian-release, BOARDFAMILY was rk3399 As per your post, I manually changed BOARDFAMILY to rockchip64. $ cat /etc/armbian-release # PLEASE DO NOT EDIT THIS FILE BOARD=helios64 BOARD_NAME="Helios64" BOARDFAMILY=rockchip64 BUILD_REPOSITORY_URL=https://github.com/armbian/build BUILD_REPOSITORY_COMMIT=173ed85 VERSION=23.08.0-trunk LINUXFAMILY=rockchip64 ARCH=arm64 IMAGE_TYPE=stable BOARD_TYPE=csc INITRD_ARCH=arm64 KERNEL_IMAGE_TYPE=Image FORCE_BOOTSCRIPT_UPDATE= VENDOR=Armbian REVISION=23.08.0-trunk BOOTSCRIPT_FORCE_UPDATE="no" BOOTSCRIPT_DST="boot.cmd" BRANCH=current That's dirty, but that let me have access to the newest armbian package releases. Can you please advise whether you know a cleaner way to match to the new BOARDFAMILY for Helios64 and its rockchip processor?

Sorry, I meant dpkg --configure -a I've got a theory on /etc/default/cpufrequtils change. Searching within openmediavault github repository for "Deploying Service Configurations ...", there is only one result : deb/openmediavault/debian/openmediavault.postinst According to openmediavault.postinst, the script echoes this message and runs omv-salt deploy run cpufrequtils 237 ######################################################################## 238 # Deploy various services when the package is installed the first time. 239 ######################################################################## 240 if [ -z "$2" ]; then 241 # Disable default nginx site. 242 nginx_dissite default || : 243 244 # Deploy the configuration for various services. 245 echo "Deploying service configurations ..." 246 LANG=C.UTF-8 omv-salt deploy run --no-color --quiet \ 247 apt cpufrequtils chrony rsyslog \ 248 watchdog monit rrdcached avahi ssh nginx collectd \ 249 phpfpm issue sysctl systemd systemd-logind || : 250 fi Thanks to https://docs.openmediavault.org/en/stable/various/advset.html#environmental-variables, I learn the configuration files used by omv-salt is stored at /srv/salt/omv Looking at the deploy stage in /srv/salt/omv I find a cpufrequtils folder kobol@helios64:~$ ls -al /srv/salt/omv/deploy/cpufrequtils/ total 20 drwxr-xr-x 3 root root 4096 Apr 5 15:58 . drwxr-xr-x 39 root root 4096 Apr 5 17:04 .. -rw-r--r-- 1 root root 2387 Mar 9 12:58 default.sls drwxr-xr-x 2 root root 4096 Apr 5 15:58 files -rw-r--r-- 1 root root 916 Mar 9 12:58 init.sls default.sls tells us the source file used by salt to configure /etc/default/cpufrequtils is files/cpufrequtils.j2 $ nano /srv/salt/omv/deploy/cpufrequtils/default.sls ... configure_default_cpufrequtils: file.managed: - name: "/etc/default/cpufrequtils" - source: - salt://{{ tpldir }}/files/cpufrequtils.j2 - template: jinja - context: cpufreq: {{ config.cpufreq }} - user: root - group: root - mode: 644 ... Finally, files/cpufrequtils.j2 calls for OMV_CPUFREQUTILS_MAXSPEED, OMV_CPUFREQUTILS_MINSPEED and OMV_CPUFREQUTILS_GOVERNOR $ nano /srv/salt/omv/deploy/cpufrequtils/files/cpufrequtils.j2 {%- set max_speed = salt['pillar.get']('default:OMV_CPUFREQUTILS_MAXSPEED', '0') -%} {%- set min_speed = salt['pillar.get']('default:OMV_CPUFREQUTILS_MINSPEED', '0') -%} {%- set governor = salt['pillar.get']('default:OMV_CPUFREQUTILS_GOVERNOR', salt['grains.filter_by']({ "default": "ondemand", "amd64": "conservative", "i386": "conservative", "armhf": "schedutil", "arm64": "schedutil", "armel": "schedutil" }, grain="osarch")) -%} {%- if cpufreq -%} ENABLE="true" {%- else -%} ENABLE="false" {%- endif %} GOVERNOR="{{ governor }}" MAX_SPEED="{{ max_speed }}" MIN_SPEED="{{ min_speed }}" The point is these values are first defined in the install script at line 543... after omv-salt is called on cpufrequtils at line 337 543 # set defaults in /etc/default/openmediavault 544 omv_set_default "OMV_CPUFREQUTILS_GOVERNOR" "${GOVERNOR}" 545 omv_set_default "OMV_CPUFREQUTILS_MINSPEED" "${MIN_SPEED}" 546 omv_set_default "OMV_CPUFREQUTILS_MAXSPEED" "${MAX_SPEED}" As per /srv/salt/omv/deploy/cpufrequtils/files/cpufrequtils.j2, there isn't any values defined yet in /etc/default/openmediavault. Thus : governor takes "schedutil" max_speed is 0 min_speed is 0

Openmediavault 7 (sandworm) is now released. Using my Helios64 with Linux 6.6.8, I followed the instructions on https://github.com/OpenMediaVault-Plugin-Developers/installScript to install OMV7. First attempt failed during the update of linux image by initramfs. The device rebooted by itself. This was due to CPUfrequency parameters that were changed in between, probably by one of OMV necessary package. Previously were : $ cat /etc/default/cpufrequtils ENABLE="true" GOVERNOR="ondemand" MAX_SPEED="1800000" MIN_SPEED="408000" And were updated to : $ cat /etc/default/cpufrequtils ENABLE="true" GOVERNOR="schedutil" MAX_SPEED="0" MIN_SPEED="0" I changed back the governor to "ondemand" using armbian-config > System > CPU. $ sudo armbian-config Needed to clean the install with : $ sudo dpkg-reconfigure -a And lauch again the install script : $ sudo curl -sSL https://github.com/OpenMediaVault-Plugin-Developers/installScript/raw/master/install | sudo bash To be sure OMV is aligned with my CPUfrequency parameters, I checked : $ cat /etc/default/openmediavault ... OMV_CPUFREQUTILS_GOVERNOR="ondemand" OMV_CPUFREQUTILS_MINSPEED="408000" OMV_CPUFREQUTILS_MAXSPEED="1800000" Those parameters were collected by the install script from /etc/default/cpufrequtils : https://github.com/OpenMediaVault-Plugin-Developers/installScript/blob/master/install Be aware OMV remove ssh-users group from /etc/ssh/sshd_config. It still possible to ssh using root. $cat /etc/ssh/sshd_config ... AllowGroups root _ssh ... As sshd_config is now controlled by OMV and may be replaced unexpectedly, you should include your user in _ssh group $ sudo usermod -a -G _ssh youruser