kantantana

More observations and progress, I get 50MB/s when doing a dd if=fileFromNfs of=/dev/null bs=1M on non-helios, and helios is the one exporting the file over nfs (from zfs). But, when pinning the nfsd process to core 4 and other sysctl changes, I get 70Mb/s, yet not every time. Other observations, with governor performance it is more stable, and higher speeds, ondemand governor introduces retries and trasmission errors, overall even a steady read of 2GB is still slower than having it run with performance governor. The other change suggested by R1 after I asked it to compare this dtb settings concerning rtl8211e was a slight adjust ment of rx_delay and tx_delay values, to 0x23, which can be done with an overlay. I can be imagining after running so many tests, but the retry counts with iperf3 is not happening as often, and if pinning iperf3 to 4 or 5, even with ondemand it is more stable and get close to 900Mbit/s. This is added to my rc.local now echo 32768 > /sys/class/net/end0/queues/rx-0/rps_flow_cnt echo 32768 > /proc/sys/net/core/rps_sock_flow_entries for i in `pgrep "nfsd"` ; do ionice -c1 -p $i ; taskset -c -p 4 $i; done

Checked the previous kernel as instructed, there is no difference, its not a regression. Setting affinity does raise it to 380Mbit/s. However, I did play with running dd if zero of to null, simultaneously with iperf3, and dd hogging 1 cpu is making it lose half of throughput, pointing to the problem being with the cpu scheduling or whatever and not with interface/driver code? Anyway, I found its good enough for now, I can achieve 800Mbit/s, but only sometimes and only if run taskset 1,2,3 iperf3, and not even every time of those 1,2,3 sometimes it has to be 4,5,6, but even the 1,2,3 times happen on retries (so it looks like 600Mbit and then 250 and then 600 again!?) And so, after many more tests, I noticed the times when I get 800Mbit/s, and other times 120Mbit/s... Becuase if I set performance governor, its then more often I can end up in reproducing 800Mbit/s, with iperf3 -A 5, or 4, but not 3 or 2... so its only the fast cpus which can give good results, the others are even if performance governor and at 1.42Ghz, still suck at 260Mbit/s Also, only with iperf3 -A 5 (affinity), I can get 0 packet-retry, other options I guess when the process is moved from cpu 0 to 1, is when retries happen, or when ondemand governor is on, if on slow cpus, retries happen when it switches freq?

I did the tests on helios64, from 2 different laptops and one desktop, between the desktops and laptops I always get 950mbit/s, it is only helios64 which gets close to 200mbit/s to any of them. They are all on the same switch and same lan. Even worse, for helios64 with iperf3 -u (udp) I get barely 1mbit/s. I did another test today, this time checked cpufreq-info helios was on 480Mhz so maybe it was too slow to increase freq? So I did dd if=/dev/zero of=/dev/null bs=1M and let it run, it was burning 1 cpu core, helios went to 1.2Ghz, did the iperf3 test and same results, barely 200Mbit/s. ethtool does say 1000 full duplex on that interface. Also tried the ethtool -k tso gso gro on and so on, no difference. The 2.5gbit/tests I did with a cable from the helios directly to an odroid h4+ which also has a 2.5gbit port, no switch inbetween at all, that test failed stability and the link was resetting on the helios64. btw this is the sha256sum a538d69ca94b31fdf91d047ec7c54970465f7d008081738baaaa5fc09be97e94 of the dtb /lib/linux-image-6.12.11-current-rockchip64/rockchip/rk3399-kobol-helios64.dtb Im using taken from this thread

@ebin-dev I used XFS before, quite liked it and benchmarked very nice numbers, but after a year or so XFS slows down to snail pace especially after reaching 70-80% space used. Havent used XFS since then, only ext4 and zfs on the nas. The cpu usage during this transfer is minimal, I tried both over NFS, over pure rsync with nothing else, and iperf3, oh and also the poor mans tar | nc transfer. All max out at 200Mbit/s (about 36mib/s) on the 1gbit interface. Surprisingly its not the sata link or zfs, because dd if=big_linux_iso of=/dev/null bs=1M count=2048 is very fast 492mib/s (its 4 drives in raidz). Just now checked that writes work, and /dev/urandom to big.file on zfs, gives 106mb/s which is low but I guess expected as writes are the speed of the slowest drive? So I have to pull out the board and solder the 2.5gbit, sigh.

Here is my dmesg about ata errors, [115174.042282] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [115174.044617] ata1.00: configured for UDMA/133 [119440.785506] ata2.00: exception Emask 0x0 SAct 0x100000 SErr 0x0 action 0x6 [119440.785537] ata2.00: waking up from sleep [119440.785554] ata2: hard resetting link [119440.785705] ata3.00: exception Emask 0x0 SAct 0x400000 SErr 0x0 action 0x6 [119440.785741] ata3.00: waking up from sleep [119440.785765] ata3: hard resetting link [119440.785813] ata4.00: exception Emask 0x0 SAct 0x80000000 SErr 0x0 action 0x6 [119440.785846] ata4.00: waking up from sleep [119440.785869] ata4: hard resetting link [119441.249776] ata4: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [119441.253555] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [119441.253794] ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [119450.515441] ata3.00: configured for UDMA/133 [119450.515514] ata3: EH complete [119450.734103] ata2.00: configured for UDMA/133 [119450.734159] ata2: EH complete [119450.763936] ata4.00: configured for UDMA/133 [119450.763985] ata4: EH complete [119450.869312] ata5.00: exception Emask 0x0 SAct 0x40 SErr 0x0 action 0x6 [119450.869329] ata5.00: waking up from sleep [119450.869338] ata5: hard resetting link [119451.337659] ata5: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [119460.873060] ata5.00: configured for UDMA/133 [119460.873117] ata5: EH complete This basically happens every time I do hdparm -Y /dev/the/drives to put them to sleep, whenever they wake up. For some reason, they wake up within seconds, oh well. The 1gbit interface sucks and gives me only 34MB/s, even with the tx on and so on, in the dmesg there is something relevant about it [ 18.692986] r8152 2-1.4:1.0: checksum fail [ 18.693009] r8152 2-1.4:1.0: unable to load firmware patch rtl_nic/rtl8156a-2.fw (-14) [ 18.744799] r8152 2-1.4:1.0 eth0: v1.12.13 [ 18.744919] usbcore: registered new interface driver r8152 [ 18.776374] r8152 2-1.4:1.0 enx646266d00ac5: renamed from eth0 edit: I fixed the rtl_nic files from the latest linux-firmware, now it can load the firmware and says # dmesg |grep r8152 [ 18.094121] usbcore: registered new device driver r8152-cfgselector [ 18.265347] r8152-cfgselector 2-1.4: reset SuperSpeed USB device number 3 using xhci-hcd [ 18.429147] r8152 2-1.4:1.0 eth0: v1.12.13 [ 18.429410] usbcore: registered new interface driver r8152 [ 18.482268] r8152 2-1.4:1.0 enx646266d00ac5: renamed from eth0 # ethtool -i enx646266d00ac5 driver: r8152 version: v1.12.13 firmware-version: rtl8156a-2 v2 04/27/23 But still, iperf3 says barely 200mbit/s, this is on the 1gbit interface. Another laptop towards same server, on same lan same switch gets 950mbit/s. I have the ondemand governor, and cpufreq settings and affinity in rc.local The 2.5gbit interface is unusable, tried many different options, but at most I can get iperf to finish once or twice, but third time it goes down to 0 and dmesg says reset. But I have only a cat 5e to test with. This helios64 doesnt have the hardware soldering fix.

root@helios64:/etc/systemd/system# uname -a Linux helios64 6.12.11-current-rockchip64 #1 SMP PREEMPT Thu Jan 23 16:23:05 UTC 2025 aarch64 GNU/Linux root@helios64:/etc/systemd/system# zfs version zfs-2.2.7-1~bpo12+1 zfs-kmod-2.2.7-1~bpo12+1 Check it out, its zfs on the latest kernel on helios64! I upgraded from the original buster back from 2021, it was running for years and had uptime more than 850 days. root@helios64:/etc/apt# cat sources.list deb http://deb.debian.org/debian/ bookworm main contrib deb-src http://deb.debian.org/debian/ bookworm main contrib deb http://security.debian.org/debian-security bookworm-security main deb-src http://security.debian.org/debian-security bookworm-security main deb http://deb.debian.org/debian/ bookworm-updates main deb-src http://deb.debian.org/debian/ bookworm-updates main deb http://deb.debian.org/debian bookworm-backports main contrib deb-src http://deb.debian.org/debian bookworm-backports main contrib The important part is the last bookworm-backports main contrib, thats where the zfs in right version comes from, also cat preferences.d/90_zfs Package: src:zfs-linux Pin: release n=bookworm-backports Pin-Priority: 990 I have the dtb from a dropbox link in here from Croatian-flag guy, and ondemand governor from 480 to 1.2ghz. Pretty stable so far, just have to set correct fan settings and drive standby time. Check your smartctl -a for drive-power-on cycles or spin-down-spin-up cycles, the stupid smartmontools smartd was causing the drives to spin up just because it was checking their values. My drives are now on 82k cycles, sigh. What power usage do you get from wall-meter, mine is on 11watt when all drives are powered down with hdparm -Y