ebin-dev

Why don't you flash linux-u-boot-edge-helios64_22.02.1_arm64 to both devices mmcblk0 and mmcblk1 and perform the python3 check after a cold start ? # messages output to the terminal while booting DDR Version 1.25 20210517 In soft reset SRX channel 0 CS = 0 MR0=0x18 MR4=0x1 MR5=0x1 MR8=0x10 MR12=0x72 MR14=0x72 MR18=0x0 MR19=0x0 MR24=0x8 MR25=0x0 channel 1 CS = 0 MR0=0x18 MR4=0x1 MR5=0x1 MR8=0x10 MR12=0x72 MR14=0x72 MR18=0x0 MR19=0x0 MR24=0x8 MR25=0x0 channel 0 training pass! channel 1 training pass! change freq to 416MHz 0,1 Channel 0: LPDDR4,416MHz Bus Width=32 Col=10 Bank=8 Row=16 CS=1 Die Bus-Width=16 Size=2048MB Channel 1: LPDDR4,416MHz Bus Width=32 Col=10 Bank=8 Row=16 CS=1 Die Bus-Width=16 Size=2048MB 256B stride channel 0 CS = 0 ...

Are you sure that you have flashed linux-u-boot-edge-helios64_22.02.1_arm64 (it contains the rockchip ddr blob) to emmc and not accidentally to sd ? emmc may be accessed as either /dev/mmcblk1p1 or as /dev/mmcblk0p1. If you started your system from emmc 'df -h' is your friend. So obviously the path has to be adapted accordingly for flashing u-boot to that device: either 'of=/dev/mmcblk1' or 'of=/dev/mmcblk0'.

Armbian bullseye images can be downloaded from the archive. And if you are having issues with OMV7 you should open another thread.

OMV 7 is not finished yet - there is just a release candidate available. You could switch to kernel 5.15.93 again but it will not resolve OMV application issues.

I did not test the procedure with more recent images. It would be very helpful if you could start from the 6.1.63 image and test it as it is (with rtl_nic firmware updated). In case of stability issues, change u-boot to the version recommended. If you are still having issues with linux 6.1.63 try linux 6.6.8, 6.1.71, or 5.15.93 instead and follow the procedure described - and please post your observations here in this thread.

I tried armbian built 6.7.4 without any modifications but it crashes almost immediately on my system. Even without hs400. The netdev watchdog message looks like the timeout caused by the mainline r8152 driver. Have never seen those events occurring before 'cut here'.

May be it is not a fault but an upstream change that leads to issues with hs400 again ? Good to hear that 6.7.x is quite stable. The remaining issue (short timeout) with the mainline r8152 driver can be reduced (eliminated) if other huge tasks do not have to compete with it. You could use taskset to assign other huge tasks to cpu5 and leave cpu4 to handle i/o.

I copied the 6.7.2 deb files yesterday (trunk-480) and will test them once I am back home next week. 6.1.71 or 5.15.93 should be fine. You could also try 6.7.2 🙂

The 6.6.8 debs were downloaded from an Armbian mirror and are not modified. Everything you need is explained here: in particular that NFS causes trouble with 6.6.x kernels and that 6.1.71 should be used instead or 5.15.93. To implement hs400 and L2 cache information you can use 'dtc'. For your convenience I attached the dtb for 6.6.8 and 5.15.93 (just copy the matching one to /boot/dtb/rockchip/rk3399-kobol-helios64.dtb). It is essential to flash the bootloader to emmc after you have installed the kernel debs, to perform a cold boot and to run 'sbc-bench -r' at least once. rk3399-kobol-helios64.dtb-5.15.93-L2-hs400 rk3399-kobol-helios64.dtb-6.6.8-L2-hs400

Newer kernel versions of the 6.6 branch (6.6.9 to 6.6.12) would not appear to run stable anymore on helios64 (6.6.8 is stable - using the ondemand governor in my use case). In kernel versions 6.6.9 to 6.6.12 energy aware scheduling (EAS) is disabled automatically - it can now only be used combination with the schedutil governor. But using schedutil is currently not a good option.

Bumping rockchip64 current to linux 6.6 (and edge to 6.7) is actually good news. But I do not think that this explains why the entire beta/pool/main folder disappeared from a mirror (but it does not matter). If nothing else is changed (keep fingers crossed), then the new Armbian built rockchip64 6.6.x and 6.7.y kernels should appear i.e. on imola.armbian.com/beta/pool/main/l/ and reappear on the mirrors.

Just to let you know: the directory 'beta/pool/main/l/linux-6.6.8/' has vanished over night (actually the entire folder fi.mirror.armbian.de/beta/pool/main). This was the location we used for downloading linux 6.6.x (6.1.y etc.) (kernel, dtb and headers). My link to the linux 6.6.8 files in this forum (as downloaded on 23.12.2023) remains active - also added links to linux 6.1.71 and 5.15.93 (dropbox).

@prahal So I used the opportunity to fix the settings for the 1G interface in armbian-hardware-optimization: My /etc/armbian-release states BOARDFAMILY=rk3399, Branch=current and I therefore edited the lines starting at line 251 - and that was successful. But I do not know if there is any positive effect on the 1G interface, since I am using 2.5G only. (I will go through the settings in /sys/class/net/end1 and see if something can be tuned for the 2.5G interface.) # cat /etc/armbian-release BOARD=helios64 BOARD_NAME="Helios64" BOARDFAMILY=rk3399 ... BRANCH=current #cat /usr/lib/armbian/armbian-hardware-optimization lines 251 ff: echo 8 > /proc/irq/$(awk -F":" "/end0/ {print \$1}" < /proc/interrupts | sed 's/\ //g')/smp_affinity echo 7 > /sys/class/net/end0/queues/rx-0/rps_cpus echo 32768 > /proc/sys/net/core/rps_sock_flow_entries echo 32768 > /sys/class/net/end0/queues/rx-0/rps_flow_cnt # systemctl status armbian-hardware-optimize.service ● armbian-hardware-optimize.service - Armbian hardware optimization Loaded: loaded (/lib/systemd/system/armbian-hardware-optimize.service; enabled; preset: enabled) Active: active (exited) since Fri 2024-01-12 15:49:01 CET; 33min ago Process: 700 ExecStart=/usr/lib/armbian/armbian-hardware-optimization start (code=exited, status=0/SUCCESS) Main PID: 700 (code=exited, status=0/SUCCESS) Tasks: 0 (limit: 4374) Memory: 2.1M CPU: 795ms CGroup: /system.slice/armbian-hardware-optimize.service I do not know if the following information is correct: # dpkg -l armbian-bsp-cli-helios64-* Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-================================-=========================================-============-===================================================== hi armbian-bsp-cli-helios64-current 23.08.0-trunk--1-PCc73b-Vccab-H1d04-B9c45 arm64 Armbian CLI BSP for board 'helios64' branch 'current'

That is true - in that script eth0 should be replaced by end0 if you wish to apply the Armbian optimisation to the 1G interface. The interrupt for xhci-hcd:usb1 devices is 46 and smb affinity is set to 10 (cpu4) on my system (used by the 2.5G interface): # cat /proc/irq/46/smp_affinity 10 # cat /proc/interrupts | grep xhci 46: 1481 0 261 0 998993 0 GICv3 142 Level xhci-hcd:usb1 93: 0 0 0 0 0 0 GICv3 137 Level dwc3-otg, xhci-hcd:usb5 CPU0 CPU1 CPU2 CPU3 CPU4 CPU5

@prahal There are many values to choose from between 1000 and 40000 (regulator-ramp-delay). Why don't you try 2000, 4000, 10000, 20000 ? (It might solve your problem)

The purpose of the 'armbian-hardware-optimization' is to bind the heavy tasks to one or both big core(s), since the little ones would not be capable of dealing with them. For some heavy tasks the kernel may need more than one big core. So I think this would make sense. Cryptosetup on a raid 5 with 5 spinning disks is definitely a huge load. Dealing with the ethernet traffic on the 2.5G interface is also a heavy load: 280Mbyts/s have to be handled in pieces of 1500bytes (MTU) - simultaneously in both directions. So it might make some sense if you modify the optimization settings depending on your use case.

This is very interesting. For regulator vdd_cpu_b, 'regulator-ramp-delay' is still set to decimal 40000 in the current dtb (6.6.8). You could try reduce that number in your dtb to increase the delay until your frequency switching program finishes its task. If the resulting value is large enough for your cpu to still respond quickly enough to tasks scheduled then you could have eliminated a source of instability. Since kernel 6.6.8 uses a more efficient scheduler you could use that one for your experiments. I actually do not think that the Kobol Team was mistaken: in their commit it is stated that the 'existing value make clock transisition time large and could causing random kernel crash'. Therefore the regulator-ramp-delay was increased from decimal 1000 to 40000 thereby decreasing the clock transition time. This was a step in the right direction - may be that one was too large ...

The r8152 driver is using xhci-hcd. The above code assigns cpu4 to those processes (echo 10 ...). I tried to assign cpu4 and cpu5 to xhci processes (echo 30) and to assign only cpu5 to ahci (sata) (echo 20), but nic offloading turned out to be more beneficial. P.S.: I am not using RAID to keep it simple (but I maintain at least 4 separate backups). The file system itself also has some impact ext4 -> btrfs -> zfs.

Thank you for trying - also very useful for others. Are you running something demanding for the hardware ? I will stay with 6.6.8 since it is 100 % stable in my use case.

@prahal It would appear that your system has some kind of hardware issue if it is not stable with linux-u-boot-edge-helios64_22.02.1_arm64.deb and kernel 5.15.93. In my use-case it is stable even with kernel 6.6.8. Regarding testing a potentially corrupt Armbian built u-boot: I am a bit reluctant to such endeavors. Helios64 is used 24/7 (by 5 people) and is not easily accessible (stored away in a rack somewhere in the basement). May be someone else could do the u-boot testing (the board on a desk would be useful) ? Otherwise I could give it a try in about 4 weeks time after I returned from some planned absence. Regarding the crash-test switching cpu frequencies: my system died after switching cpu frequencies about 580 times (in less than a second), with linux-u-boot-edge-helios64_22.02.1_arm64.deb on kernel 6.6.8 (see the attached log). Looking at the output of cpufreq-info it can be seen which cpu-frequency states are used most often. My system normally almost exclusively jumps between 600MHz <-> 1.8GHz (big cores) and between 408MHz <-> 600MHz or between 400MHz <-> 1.42GHz (little cores). The only thing I did in that context was running sbc-bench -r which supposedly changed some performance related settings permanently. I think that omitting the intermediate states reduces switching between states and thus enhances responsiveness and stability while reducing the burden on the scheduler. I don't know if this helps, but I attached the cpu frequency transition tables for cpu5 and cpu0 (after about 3h uptime) # cat /sys/devices/system/cpu/cpu5/cpufreq/stats/trans_table From : To : 408000 600000 816000 1008000 1200000 1416000 1608000 1800000 408000: 0 0 0 0 0 0 0 0 600000: 0 0 140 13 7 7 1 1126 816000: 0 130 0 13 3 1 2 48 1008000: 0 15 18 0 4 2 2 1 1200000: 0 5 6 7 0 9 3 7 1416000: 0 3 3 4 10 0 15 9 1608000: 0 2 1 1 8 18 0 18 1800000: 0 1139 29 4 5 7 25 0 # cat /sys/devices/system/cpu/cpu0/cpufreq/stats/trans_table From : To : 408000 600000 816000 1008000 1200000 1416000 408000: 0 1133 14 9 3 1002 600000: 1081 0 5 3 2 134 816000: 12 6 0 46 3 10 1008000: 7 2 44 0 11 21 1200000: 1 4 6 17 0 28 1416000: 1061 79 8 10 37 0 cpufreq-switching-2-b.log

Good news! Did you modify the dtb to support emmc hs400 speed and l2-cache information ? It would be nice if you could also try the 6.1.70 kernel.

@prahal Linux 6.6.8 and linux-u-boot-edge-helios64_22.02.1_arm64 is used since December 23rd without any Linux oops (despite the NETDEV Watchdog having to reset occasionally the mainline r8152 driver during iperf3 stress tests - but not during operation). @alchemist observed however, that NFS causes issues with 6.6.8 but not with 6.1.70 but that would not appear to be Helios64 specific. My use case: 24/7 as a DNS server, file server, nextcloud server, music server, plex server, and for home automation - kept everything simple (i.e. ext4 file system, no NFS).

Which u-boot version do you refer to (the one including pull requests from @prahal) ? P.S.: I included a note that in case you use NFS, kernel 6.1.70 should be used.

@OdyX After executing sbc-bench I switched the governor back to ondemand. I never had issues with it. You may try schedutil too. I added a download link (above in this thread) to the dtb enhanced with hs400 and l2 cache information (for linux 6.6.8). You can safely copy it to /boot/dtb/rockchip/rk3399-kobol-helios64.dtb. Difficult to say if 'linux-cpupower' caused that trouble. # cat /etc/default/cpufrequtils ENABLE=true MIN_SPEED=408000 MAX_SPEED=1800000 GOVERNOR=ondemand

In my use case there are no issues so far - using it 24/7 as a DNS server, file server, nextcloud server, music server, plex server, and for home automation. I kept everything simple (i.e. ext4 file system, no NFS). @OdyX The undocumented errors you describe are hard to believe. If your system really behaves like that you should consider to donate your board for testing and buy something else. Otherwise: flash u-boot version 21.08.9. If that does not help, you can easily change the linux kernel: try 5.15.93. Just install linux-image, linux-headers, linux-dtb with dpkg -i linux* and do not forget to delete the remaining 6.6.8 links from / and from /boot. And make sure that you have the right u-boot flashed to emmc.