ebin-dev

@prahal Current kernel 6.9.6 works just fine on Helios64 (headless) thanks to your PR. I am still using linux-u-boot-edge-helios64_22.02.1_arm64. Level2 cache-info is now included: # lscpu -C NAME ONE-SIZE ALL-SIZE WAYS TYPE LEVEL SETS PHY-LINE COHERENCY-SIZE L1d 32K 192K 4 Data 1 128 64 L1i 32K 224K 2 Instruction 1 256 64 L2 512K 1.5M 16 Unified 2 512 64 The modified dtb (hs400, modified opp-table-1) is attached below (copy to /boot/dtb/rockchip/rk3399-kobol-helios64.dtb ; execute 'update-initramfs -u'). P.S.: sata performance needs to be addressed and helios64-heartbeat-led.service is not working rk3399-kobol-helios64.dtb-6.9.6-L2-hs400-opp

Since linux-u-boot-edge-helios64_22.02.1_arm64 does'nt seem to be downloadable anymore from Armbian servers, I added a download link to it from my dropbox account.

@prahal There are two versions of the bootloader in the repository (edge and current). I just tried the current u-boot from the repository. It works but dmesg outputs seven of the following messages: [ 9.674633] rockchip-i2s ff8a0000.i2s: Could not register PCM This note mentions that such error messages could mean that DMA and sound are missing in the linux kernel, could this be ignored ? P.S.: Those messages are not thrown if the recommended bootloader (linux-u-boot-edge-helios64_22.02.1_arm64) is flashed to emmc.

Checking that note reveals: Note: at the current status, helios64 board patches have been disabled because the base device tree does not apply on kernel 6.9. So helios64 is cancelled as of linux 6.9.x until someone takes care of the issue. Thank you very much.

MergerFS is on github - there is a section 'systemd mount' in the ReadMe with an mergerfs.service example...

You probably just have to change the systemd service file nfs-server.service: in its [UNIT] section add the mergerFs service file (see also here) : # nfs-server.service # replace 'mergerfs.service' by the name of your MergerFS service file [Unit] After=mergerfs.service Requires=mergerfs.service

@specs A current 6.9.0 edge kernel (collabora version) is availale on beta.armbian.com (linux-dtb and linux-headers are available too). They can be downloaded to the filesystem using 'wget <URL>' and installed via 'dpkg -i linux*' (without armbian-config). After updating Armbian 24.2.1 (minimal) from linux 5.10.160 to 6.9.0: (system on emmc, data on a 4TB sata ssd (m.2)) ____ _ ____ ____ | _ \ ___ ___| | __ | ___|| __ ) | |_) / _ \ / __| |/ / |___ \| _ \ | _ < (_) | (__| < ___) | |_) | |_| \_\___/ \___|_|\_\ |____/|____/ Welcome to Armbian 24.2.1 Bookworm with Linux 6.9.0-collabora-rockchip-rk3588 No end-user support: community creations System load: 1% Up time: 27 min Memory usage: 3% of 7.35G IP: 192.168.xx.31 CPU temp: 45°C Usage of /: 33% of 14G I tried a network performance benchmark and it rocks: 2x 2.34 Gbits/s (bidirectional) ! (armbian-hardware-optimize disabled) # ./iperf3 -c 192.168.xx.31 -p 5201 (client -> rock 5b) Connecting to host 192.168.xx.31, port 5201 [ 5] local 192.168.xx.54 port 63943 connected to 192.168.xx.31 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.01 sec 284 MBytes 2.37 Gbits/sec [ 5] 1.01-2.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 2.00-3.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 3.00-4.00 sec 281 MBytes 2.35 Gbits/sec [ 5] 4.00-5.00 sec 281 MBytes 2.35 Gbits/sec [ 5] 5.00-6.01 sec 282 MBytes 2.35 Gbits/sec [ 5] 6.01-7.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 7.00-8.00 sec 281 MBytes 2.35 Gbits/sec [ 5] 8.00-9.00 sec 281 MBytes 2.35 Gbits/sec [ 5] 9.00-10.00 sec 280 MBytes 2.35 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 2.74 GBytes 2.36 Gbits/sec sender [ 5] 0.00-10.01 sec 2.74 GBytes 2.35 Gbits/sec receiver # ./iperf3 -c 192.168.xx.31 -p 5201 -R (rock 5b -> client) Connecting to host 192.168.xx.31, port 5201 Reverse mode, remote host 192.168.xx.31 is sending [ 5] local 192.168.xx.54 port 63945 connected to 192.168.xx.31 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.01 sec 279 MBytes 2.33 Gbits/sec [ 5] 1.01-2.01 sec 281 MBytes 2.35 Gbits/sec [ 5] 2.01-3.01 sec 280 MBytes 2.35 Gbits/sec [ 5] 3.01-4.01 sec 281 MBytes 2.35 Gbits/sec [ 5] 4.01-5.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 5.00-6.01 sec 281 MBytes 2.35 Gbits/sec [ 5] 6.01-7.01 sec 280 MBytes 2.35 Gbits/sec [ 5] 7.01-8.01 sec 281 MBytes 2.35 Gbits/sec [ 5] 8.01-9.01 sec 280 MBytes 2.35 Gbits/sec [ 5] 9.01-10.01 sec 281 MBytes 2.35 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.01 sec 2.74 GBytes 2.35 Gbits/sec 2 sender [ 5] 0.00-10.01 sec 2.74 GBytes 2.35 Gbits/sec receiver #./iperf3 -c 192.168.xx.31 -p 5201 --bidir (bidirectional) Connecting to host 192.168.xx.31, port 5201 [ 5] local 192.168.xx.54 port 63948 connected to 192.168.xx.31 port 5201 [ 7] local 192.168.xx.54 port 63949 connected to 192.168.xx.31 port 5201 [ ID][Role] Interval Transfer Bitrate [ 5][TX-C] 0.00-1.00 sec 278 MBytes 2.33 Gbits/sec [ 7][RX-C] 0.00-1.00 sec 273 MBytes 2.29 Gbits/sec [ 5][TX-C] 1.00-2.01 sec 280 MBytes 2.34 Gbits/sec [ 7][RX-C] 1.00-2.01 sec 281 MBytes 2.35 Gbits/sec [ 5][TX-C] 2.01-3.00 sec 278 MBytes 2.34 Gbits/sec [ 7][RX-C] 2.01-3.00 sec 279 MBytes 2.35 Gbits/sec [ 5][TX-C] 3.00-4.01 sec 280 MBytes 2.34 Gbits/sec [ 7][RX-C] 3.00-4.01 sec 281 MBytes 2.35 Gbits/sec [ 5][TX-C] 4.01-5.01 sec 279 MBytes 2.34 Gbits/sec [ 7][RX-C] 4.01-5.01 sec 280 MBytes 2.35 Gbits/sec [ 5][TX-C] 5.01-6.01 sec 279 MBytes 2.34 Gbits/sec [ 7][RX-C] 5.01-6.01 sec 280 MBytes 2.35 Gbits/sec [ 5][TX-C] 6.01-7.01 sec 279 MBytes 2.34 Gbits/sec [ 7][RX-C] 6.01-7.01 sec 280 MBytes 2.35 Gbits/sec [ 5][TX-C] 7.01-8.00 sec 278 MBytes 2.34 Gbits/sec [ 7][RX-C] 7.01-8.00 sec 279 MBytes 2.35 Gbits/sec [ 5][TX-C] 8.00-9.01 sec 280 MBytes 2.34 Gbits/sec [ 7][RX-C] 8.00-9.01 sec 281 MBytes 2.35 Gbits/sec [ 5][TX-C] 9.01-10.01 sec 279 MBytes 2.34 Gbits/sec [ 7][RX-C] 9.01-10.01 sec 280 MBytes 2.35 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID][Role] Interval Transfer Bitrate Retr [ 5][TX-C] 0.00-10.01 sec 2.72 GBytes 2.34 Gbits/sec sender [ 5][TX-C] 0.00-10.01 sec 2.72 GBytes 2.34 Gbits/sec receiver [ 7][RX-C] 0.00-10.01 sec 2.73 GBytes 2.35 Gbits/sec 1 sender [ 7][RX-C] 0.00-10.01 sec 2.73 GBytes 2.34 Gbits/sec receiver

Sorry - I am probably the only one who is interested in network performance of the 2.5G interface (theoretical max 2.35Gbit/s). (iperf 3.17.1 measurements attached)

Just a remark: setting MIN_SPEED to 600 MHz would have NO effect on power consumption but required frequency switching would be less: switching between 408 and 600MHz would be avoided and the system could transition directly from 600 MHz to the highest frequency (essentially switching only between two states for each cpu). The huge power savings are impressive!

Due to its current status I have disabled armbian-hardware-optimization (some 'volunteers' are needed to fix it) and run the following code in /etc/rc.local instead. Alternatively you could seek the line 'echo 200000' in armbian-hardware-optimization and change the lines around it accordingly. @SIGSEGV Yes - thereby you set the sampling_rates per cpu cluster to 40000 and 51000 on the current kernel (actually by the line starting with 'echo $(cat ...' ). #!/bin/sh -e # # rc.local # # This script is executed at the end of each multiuser runlevel. # Make sure that the script will "exit 0" on success or any other # value on error. # # In order to enable or disable this script just change the execution # bits. # # By default this script does nothing. cd /sys/devices/system/cpu/cpufreq for cpufreqpolicy in 0 4 ; do echo 1 > policy${cpufreqpolicy}/ondemand/io_is_busy echo 25 > policy${cpufreqpolicy}/ondemand/up_threshold echo 10 > policy${cpufreqpolicy}/ondemand/sampling_down_factor echo $(cat policy${cpufreqpolicy}/cpuinfo_transition_latency) > policy${cpufreqpolicy}/ondemand/sampling_rate done for i in $(awk -F":" "/ahci/ {print \$1}" < /proc/interrupts | sed 's/\ //g'); do echo 30 > /proc/irq/$i/smp_affinity done for i in $(awk -F":" "/xhci/ {print \$1}" < /proc/interrupts | sed 's/\ //g'); do echo 20 > /proc/irq/$i/smp_affinity done exit 0

Actually the changes to the dtb were figured out and were proposed by @prahal and therefore it would be up to Prahal to submit a PR. From my perspective there is nothing against a PR right now as the changes are sufficiently tested and have a very positive effect on stability.

There are new Armbian 24.05 images available on the Helios64 download page: both images Bookworm minimal and Jammy Desktop are based on linux 6.6.30 (download them !). Again, the rtl_nic firmware (in /lib/firmware/rtl_nic) should be replaced by the version downloaded from git.kernel.org, such that the 2.5G LAN interface works correctly. I would also recommend to copy the dtb attached below to /boot/dtb/rockchip/rk3399-kobol-helios64.dtb (execute 'update-initramfs -u' after that). It includes the 75mV bump of the opp states for the fast cores as suggested by @prahal, it enables the L2 cache info and it enables hs400 speed on emmc again. In particular the 75mV bump has a very positive effect on stability. The bootloader that comes with it would appear to contain the Rockchip DDR blob. It should be fine. If you have an issue with u-boot, just flash linux-u-boot-edge-helios64_22.02.1_arm64 as recommended before. The cpufreq ondemand governor is still the best choice. Good settings are # cat /etc/default/cpufrequtils ENABLE=true MIN_SPEED=600000 MAX_SPEED=1800000 GOVERNOR=ondemand Enjoy. P.S.: If you like a system more responsive to server tasks or push the 2.5G interface to the limits, some fine tuning is helpful. rk3399-kobol-helios64.dtb-6.6.30-L2-hs400-opp

Thanks for the initiative with the PR. Here is what happens: The cpufreqpolicy sets a generic value for the sampling_rate of 200000 (this is how often the governor’s worker routine should run, in microseconds). The sampling_rate should be set to a nominal value close to the cpuinfo_transition_latency (49000 and 40000 nanoseconds for the clusters of rk3399) such that it is effectively about 1000 times as large. For rk3399 boards the cluster sampling_rates of 49000 (big) and 40000 (little) are very responsive and not too demanding for the cpus. In that case no problems occur if io_is_busy is set to 1 ! I added a comment to PR6507 and requested that the sampling_rate should be a parameter (per cpu cluster) and be set to the value of the cpuinfo_transition_latency of that cpu. The fine tuned values for Helios64 are those (io_is_busy set to 1 again, sampling_rate set to more responsive values): for cpufreqpolicy in 0 4 ; do echo 1 > /sys/devices/system/cpu/cpufreq/policy${cpufreqpolicy}/ondemand/io_is_busy echo 25 > /sys/devices/system/cpu/cpufreq/policy${cpufreqpolicy}/ondemand/up_threshold echo 10 > /sys/devices/system/cpu/cpufreq/policy${cpufreqpolicy}/ondemand/sampling_down_factor echo $(cat /sys/devices/system/cpu/cpufreq/policy${cpufreqpolicy}/cpuinfo_transition_latency) > /sys/devices/system/cpu/cpufreq/policy${cpufreqpolicy}/ondemand/sampling_rate done Edit: With these changes the timeout issues of the 2.5G interface are resolved for 6.6.29 and 6.6.30 too. Helios64 is one of the best arm based NAS systems I have seen so far. I am not going to replace it anytime soon.

@SteeMan I tested the latest changes in armbian-hardware-optimize with linux 6.6.8, 6.6.29 and 6.6.30 on a Helios64. The above cpufreqpolicy does not work if 'io_is_busy' is set to 1 on Helios64 at least if the 2.5G interface is used (transmit queue timeouts). In armbian-hardware-optimize (a-h-o) it is set to 1. It must be set to 0 (kernel default). If the sampling_rate is set to a more responsive value of 50000 the issues vanish with io_is_busy set to 1. The optimizations for the 1G interface (eth0/end0) in the armbian script (see below) have the effect that those tasks are not performed in the NIC anymore (hardware) but on the CPUs in software. The effect is a huge additional load on the CPUs. Anyway such settings are not necessary with the new EEVDF scheduler anymore. The reason why I am disabling armbian-hardware-optimize is that I need a stable system and I want to avoid any random and untested changes on my system. echo 8 > /proc/irq/$(awk -F":" "/eth0/ {print \$1}" < /proc/interrupts | sed 's/\ //g')/smp_affinity echo 7 > /sys/class/net/eth0/queues/rx-0/rps_cpus echo 32768 > /proc/sys/net/core/rps_sock_flow_entries echo 32768 > /sys/class/net/eth0/queues/rx-0/rps_flow_cnt

TLDR; The dtb contains a 75 mV bump for all fast cores as suggested by @prahal. You can find the opp-table-1 in this thread above (posted April 19). Use device tree compiler 'dtc' to apply your own changes (link). The dtb further contains a fix that enables hs400 speed on emmc also proposed by prahal. There is also a change for providing L2 cache information (this is all explained in this post) - and there is also a note that 6.6.29 is stable with those modifications. However, 6.6.29 and 6.6.30 bugs me with frequent transmit queue timeouts while using the 2.5G interface (NETDEV WATCHDOG: end1 (r8152): transmit queue 0 timed out 8584 ms). I could not find any combination of settings to get rid of this. Therefore I switched back to 6.6.8 - it works perfectly fine - absolutely no issues. I am on linux 6.6.30 with the following configuration were armbian-hardware-optimize.service is disabled and NAS performance is restored: (The transfer rate from Helios64 to a MacBook Pro is 280MByte/s using a 2.5G switch in between and a 2.5G USB-C Dongle (StarTech US2GC30 (rtl8156bg)) attached to the Mac) root@helios64:~# cat /etc/default/cpufrequtils ENABLE=true MIN_SPEED=600000 MAX_SPEED=1800000 GOVERNOR=ondemand root@helios64:~# cat /etc/rc.local #!/bin/sh -e # # rc.local # # This script is executed at the end of each multiuser runlevel. # Make sure that the script will "exit 0" on success or any other # value on error. # # In order to enable or disable this script just change the execution # bits. # # By default this script does nothing. for cpufreqpolicy in 0 4 ; do echo 1 > /sys/devices/system/cpu/cpufreq/policy${cpufreqpolicy}/ondemand/io_is_busy echo 25 > /sys/devices/system/cpu/cpufreq/policy${cpufreqpolicy}/ondemand/up_threshold echo 10 > /sys/devices/system/cpu/cpufreq/policy${cpufreqpolicy}/ondemand/sampling_down_factor echo $(cat /sys/devices/system/cpu/cpufreq/policy${cpufreqpolicy}/cpuinfo_transition_latency) > /sys/devices/system/cpu/cpufreq/policy${cpufreqpolicy}/ondemand/sampling_rate done for i in $(awk -F":" "/xhci/ {print \$1}" < /proc/interrupts | sed 's/\ //g'); do echo 20 > /proc/irq/$i/smp_affinity done for i in $(awk -F":" "/ahci/ {print \$1}" < /proc/interrupts | sed 's/\ //g'); do echo 30 > /proc/irq/$i/smp_affinity done exit 0 root@helios64:~# systemctl disable armbian-hardware-optimize.service root@helios64:~# reboot Edit: 'sampling_rate' reduced, it was not responsive enough - now 49000 (big cores) and 40000 (little cores) in microseconds. The nominal value is the same as 'cpuinfo_transition_latency'. If you are happy with 6.6.29 or 6.6.30 go ahead. For your convenience I attach the modified dtbs. rk3399-kobol-helios64.dtb-6.6.8-L2-hs400-opp rk3399-kobol-helios64.dtb-6.6.29-L2-hs400-opp rk3399-kobol-helios64.dtb-6.6.30-L2-hs400-opp

I gave up on 6.6.29/30: in general they are stable but the 2.5G interface causes trouble again (NETDEV WATCHDOG: end1 (r8152): transmit queue 0 timed out 8584 ms) although the code for the r8152 driver did not change since the release of 6.6.8. Therefore I am back again on 6.6.8. It runs fine with the kernel defaults, irq smp_affinity set for ahci and xhci (see above) (armbian-hardware-optimize.service disabled) and it also benefits from the 75 mV opp increase (dtb attached). rk3399-kobol-helios64.dtb-6.6.8-L2-hs400-opp

You are running out of stack space during rsync.

It is not only about setting default cpufreq governors - there are also other settings essential for the network performance (see here). And for Helios64 it is essential to set the irq smp_affinity: for i in $(awk -F":" "/xhci/ {print \$1}" < /proc/interrupts | sed 's/\ //g'); do echo 20 > /proc/irq/$i/smp_affinity done for i in $(awk -F":" "/ahci/ {print \$1}" < /proc/interrupts | sed 's/\ //g'); do echo 30 > /proc/irq/$i/smp_affinity done If armbian-hardware-optimization is disabled or has vanished at least the irq smp_affinity should be set for sata (ahci) and for the 2.5G LAN interface (xhci) (i.e. in /etc/rc.local)

I am using the default values (after various tests). # cat /etc/default/cpufrequtils ENABLE=true MIN_SPEED=408000 MAX_SPEED=1800000 GOVERNOR=ondemand Edit: I would love to use schedutil but low priority tasks take three times as long with it to complete compared to ondemand (!!) (I tested that with a complete file scan of my nextcloud installation). There was a lot discussion going on in armbian/build on github. Finally the Armbian default governor was set to schedutil again instead to ondemand. But schedutil parameters need to be fine-tuned such that EAS works correctly and from my own observations I can confirm that this is clearly not (yet) the case at least for the rk3399-kobol-helios64.dtb. Anyway - you can set min_speed to 600000 for all cores (ondemand): it does not affect power consumption at all, but it renders your Helios64 more responsive. And disable armbian-hardware-optimize - it degrades performance.

I am using the ondemand governor. # cat /etc/default/cpufrequtils ENABLE=true MIN_SPEED=408000 MAX_SPEED=1800000 GOVERNOR=ondemand

Just download the file "rk3399-kobol-helios64.dtb-6.6.29-L2-hs400-opp" and copy it to /boot/dtb/rockchip/rk3399-kobol-helios64.dtb and reboot. It replaces the original rk3399-kobol-helios64.dtb which you may rename to rk3399-kobol-helios64.dtb-orig beforehand. It clearly enhances the stability of the system - but it cannot resolve any application errors.

Did you increase opp voltages by 75mV for the fast cores in this test ? (dtb attached) rk3399-kobol-helios64.dtb-6.6.29-L2-hs400-opp

Current kernel 6.6.29 uses the lowest frequencies 400 and 600 MHz also for the big cores, contrary to linux 6.6.8, where the lowest frequency used by the big cores is 800 MHz. So I would assume from what you say that linux 6.6.29 should be more stable. However, I was puzzled why this is not the case with my system: actually none of the kernels 6.6.x (x>8) was stable on my system until I tried 6.6.29 with the 75mV bump for all states of the big cores ! The explanation I have for that is that the jump from the lowest state 408 to 1.8 GHz is somewhat larger than from 800 MHz and that the correct voltage feeded to the cores becomes more critical in those cases. So with the 75 mV bump for all states of the big cores, that source of instability seems to have vanished now: 6.6.29 is stable so far (at least on my system). # cat /etc/default/cpufrequtils ENABLE=true MIN_SPEED=408000 MAX_SPEED=1800000 GOVERNOR=ondemand # transition tables, linux 6.6.29 # cat /sys/devices/system/cpu/cpufreq/policy4/stats/trans_table From : To : 408000 600000 816000 1008000 1200000 1416000 1608000 1800000 408000: 0 1705 141 1 5 1 1 394 600000: 1695 0 7 2 0 3 0 120 816000: 137 8 0 2 3 3 2 21 1008000: 2 1 5 0 0 0 0 1 1200000: 5 1 2 0 0 2 3 4 1416000: 2 0 3 0 3 0 3 4 1608000: 1 0 0 1 1 3 0 5 1800000: 407 111 18 3 5 3 2 0 # cat /sys/devices/system/cpu/cpufreq/policy0/stats/trans_table From : To : 408000 600000 816000 1008000 1200000 1416000 408000: 0 2951 7 3 2 1547 600000: 2865 0 2 2 2 139 816000: 8 2 0 1 1 7 1008000: 4 0 1 0 4 4 1200000: 1 1 1 1 0 21 1416000: 1633 55 8 6 16 0

Linux 6.6.29 is stable on my system so far when used with the modified dtb (in particular applying the 75mV bump for the big cores, dtb attached below). Could you give it a try ? rk3399-kobol-helios64.dtb-6.6.29-L2-hs400-opp

There are only a few frequencies involved - at least on my board (see the transition tables in my other message in the parallel thread). Upping the voltage by 75mV - as you suggested - helped my board to get rid of remaining few occasional issues during the boot process ! I am measuring the combined power consumption of Helios64 and two 2.5G switches (together 12.66W idle) - it does not seem to be affected at all by that small change. Temperature reported by the Armbian welcome screen is 44 °C (ambient temp 19°C in the basement). Is the remaining crash - once a day on your Helios64 - positively affected by upping the voltage to 1.2V for all states ? If not it may be caused by something else. Edit: I am currently testing 6.6.29 using a modified dtb (L2, hs400, opp up by 75mV)