ebin-dev

Did you realize that none of your USB Ports are accessible anymore (dwc3 usb regression) ? This is the topic of the parallel thread. There are many things that can go wrong if you try userspace upgrades. You should set up your system starting from Armbian 23.05 - 6.1.36 image (bookworm).

Do not forget to disable the Armbian updates and to copy the firmware files to /lib/firmware/rtl_nic. And thank you too.

Linux 6.6 was released yesterday and it is now another option: it boots just fine on Helios64 (also off emmc) and I am using it now instead of 6.1.60. Edit: Download Link: Linux 6.6 debs

There is an alternative way to install non-free firmware using apt. But yes - you can copy the content of the rtl_nic folder downloaded from git.kernel.org into /lib/firmware/rtl_nic/.

@prahal Regarding the remaining glitch (red sata LEDs flash one after the other during boot): I observe it too with linux 6.6-rc7. To exclude that this is u-boot related: which version of u-boot do you use (on sd/emmc) ? (A stock image ?) Edit: My system also had the "free() invalid pointer" issue and I repaired it by flashing a new bootloader (linux-u-boot-edge-helios64_22.02.1_arm64 , as discussed here).

There are alternatives - you can install all the non-free firmware at once using apt. But yes - you just need to copy the content of the rtl_nic folder downloaded from git.kernel.org to /lib/firmware/rtl_nic in your filesystem (at least the 9 firmware files indicated above).

@prahal I transferred the system from sd to emmc using armbian-config and once the system booted from emmc I had kernel crashes. And booting from sd was really slow after that. In fact I had to set up the entire system again from scratch. The second try was as described in my previous post (rsync sd to formatted emmc with excludes). I would assume that the bootloader on sd was overwritten and I have no idea what went wrong with the emmc. I use it 24/7 now (bookworm 12.2 with linux 6.1.60 on emmc). Would you like to look at the remaining glitch that I observe with linux 6.1.60 during boot (using a spare sd with bookworm on it) ? The sata bus is rescanned during boot and the red sata 1-5 LEDs flash one after the other at the time when the heartbeat LED starts to blink. This was not the case with linux 6.1.36. dmesg.txt

In order to migrate to bookworm the tests on SD worked fine (after the dwc3 usb issue was solved) but the trouble began after it was transferred from sd to emmc. I used armbian-config for that and I was about to loose my data ... My starting point was the Armbian 23.05 - 6.1.36 Image and I am using linux 6.1.60 now, since the header files were missing in the Armbian Image I had to compile them myself. As far as I understand, this is the new Armbian policy ... Just formatting the emmc partition and to rsync the data from sd to emmc using a backup script worked for me (fstab needs adaptation and some folders have to be recreatad in /var/log with proper access rights i.e. cups, nginx, mysql ...). There are no read/write errors and I am using bookworm on emmc now for about a week now. In fact it is reliable. Also the network operates amazingly fast. The only remaining issue is: while the heartbeat LED starts to operate, the red LEDs on the front panel briefly light up (sata1 to sata5, bus rescan) and the fans spin up for a few seconds , then turn to normal operation. Could this be u-boot related ? Would you have an idea ? (see the parallel thread)

There are other driver versions for the realtek r8152 driver. Just out of curiosity I tried version r8152-v2.16.3 and r8152-v2.17.1. Both drivers crash immediately once a large data transfer is executed at least via the 2.5G interface (r8152 2-1.4:1.0 end1: Tx timeout; xhci-hcd xhci-hcd.0.auto: bad transfer trb length 16741766 in event trb). It would appear that those two driver versions do not patch the firmware during boot like the mainline driver (r8152 2-1.4:1.0: load rtl8156a-2 v2 04/27/23 successfully). The bottom line is, that we have stable bullseye image with linux-6.1.36 (without header files) and a stable kernel 6.1.60 (with header files) both using the mainline r8152 driver (the required r8152 firmware files have to be in /lib/firmware/rtl_nic ; can be downloaded from here). Linux 6.1.60 has a remaining glitch to be solved, since during boot there is an unnecessary rescan of the bus and the fans spin up for a second. If anyone has a hint that helps to solve this, I would be grateful (dmesg output attached). Cheers P.S. The Armbian images on the Helios64 download page (version Aug. 31, 2023) are still broken, no usb support due to the dwc3 regression. P.S.': iperf3 measurements between Helios64 (server) and a client through the 2.5G interface 😀: ./iperf3 -c 192.168.xx.xx -p 5201 Connecting to host 192.168.xx.xx, port 5201 [ 5] local 192.168.xx.xx port 57314 connected to 192.168.xx.xx port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 275 MBytes 2.31 Gbits/sec [ 5] 1.00-2.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 2.00-3.00 sec 281 MBytes 2.36 Gbits/sec [ 5] 3.00-4.00 sec 281 MBytes 2.36 Gbits/sec [ 5] 4.00-5.00 sec 281 MBytes 2.36 Gbits/sec [ 5] 5.00-6.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 6.00-7.00 sec 281 MBytes 2.36 Gbits/sec [ 5] 7.00-8.00 sec 278 MBytes 2.34 Gbits/sec [ 5] 8.00-9.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 9.00-10.00 sec 281 MBytes 2.36 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 2.73 GBytes 2.35 Gbits/sec sender [ 5] 0.00-10.00 sec 2.73 GBytes 2.35 Gbits/sec receiver dmesg.txt

I am on linux kernel 6.1.60 now and both 1G and 2.5G LAN interfaces operate with the maximum expected throughput (about 300 MByte/s on the 2.5G interface) - without any further patches applied. If you like to test them, you can download the debs (kernel, header, dtb) from here ( install with 'dpkg -i linux*'). There is one little glitch: while the heartbeat LED starts to operate, the red LEDs on the front panel briefly light up (sata1 to sata5, bus rescan) and the fans spin up for a few seconds , then turn to normal operation. P.S.: dmesg output attached, 2.5G interface used The usb 2-1.4 interface (Realtek USB 10/100/1G/2.5G LAN) is reset 2 times by the driver during boot and it works flawlessly after that. I consider this normal. bus-scan-delay does not help to get rid of the bus rescan. I tried a delay of 2s: [ 2.832048] rockchip-pcie f8000000.pcie: wait 2000 ms (from device tree) before bus scan dmesg.txt

Bookworm (with linux 6.1.58 (patched) incl. header files) is now booting from emmc and it is indeed absolutely stable using the standard settings. To do that I just formatted the emmc partition and transferred the files from sd to emmc using a backup script. /etc/fstab needs the correct UUID of the emmc (displayed by 'blkid') and some folders (i.e. cups, mysql, redis, nginx, ...) may have to be manually created in /var/log with proper access rights. Make a copy of your sd card before, if you prefer armbian-config to do that (boot from emmc - system on emmc) as the bootloader on SD may be negatively affected after that. Will now test different rtl8156 drivers and see if there is any effect on performance and stability of the 2.5G LAN interface for current kernels (>= 6.1.59).

@alchemist USB ports are back - but the 2.5G interface is now unstable: transferring a 2GB file (Speed: 2500Mb/s) is interrupted (r8152 Tx timeout) and the interface is restarted by the watchdog process. Could you test transferring large files through the 2.5G interface set to speed 2500Mb/s ? I switched back to the previous kernel 6.1.58 (patched) - no TX timeouts ...

dwc3 was patched In the current linux kernel 6.1.59 and the dwc3 error disappeared. All USB devices are accessible again. Now a new Armbian build based on linux 6.1.59 should be pushed to the Helios64 download page.

If you intend to switch to bookworm, the image Armbian_23.5.4_Helios64_bookworm_current_6.1.36.img is indeed a good starting point: I have tested the 1G and 2.5G LAN interfaces and they perform well out of the box. On my Helios64, I could not obtain any stability issues with this system using kernel 6.1.36 (I have soldered the cable to the motherboard as suggested on the kobol.io website). As mentioned above, the dwc3 usb regression ist not jet present in that kernel. So I would advise to use that image if you intend to upgrade to bookworm. However, I could not obtain any matching header files for that kernel version which are necessary if the dkms is needed (on my system by some home automation software). Therefore, I had to compile the current linux kernel 6.1.58 (linux-image, linux-headers, linux-dtb) while applying a patch to solve the dwc3 usb issue (patch is discussed here). Welcome to Armbian 23.08.0-trunk Bookworm with Linux 6.1.58-current-rockchip64 No end-user support: community creations System load: 2% Up time: 2:48 Memory usage: 30% of 3.77G IP: xx.xx.xx.xx CPU temp: 47°C Usage of /: 14% of 58G storage/: 61% of 3.6T storage temp: 25°C All USB devices are working now - also with the current kernel (patched). The driver for the realtek 2.5G interface (mainline version v1.12.13 - 'modinfo r8152') performs well if all the following firmware files are present in /lib/firmware/rtl_nic/ (can be downloaded from here): # ls -la /lib/firmware/rtl_nic/ drwxrwxr-x 2 root root 4096 Oct 18 16:56 . drwxr-xr-x 27 root root 12288 Oct 16 15:46 .. -rw-rw-r-- 1 root root 3328 Oct 16 15:32 rtl8125b-2.fw -rw-r--r-- 1 root root 1768 Oct 16 15:32 rtl8153a-2.fw -rw-r--r-- 1 root root 1440 Oct 16 15:32 rtl8153a-3.fw -rw-r--r-- 1 root root 712 Oct 16 15:32 rtl8153a-4.fw -rw-rw-r-- 1 root root 1880 Oct 16 15:32 rtl8153b-2.fw -rw-r--r-- 1 root root 832 Oct 16 15:32 rtl8153c-1.fw -rw-r--r-- 1 root root 4024 Oct 16 15:32 rtl8156a-2.fw -rw-r--r-- 1 root root 7256 Oct 16 15:32 rtl8156b-2.fw -rw-rw-r-- 1 root root 976 Oct 16 15:32 rtl8168h-2.fw # cat /var/log/syslog | grep r8152 2023-10-19T07:01:02.167978+02:00 helios64 kernel: [ 6.672719] r8152 2-1.4:1.0: load rtl8156a-2 v2 04/27/23 successfully 2023-10-19T07:01:02.167981+02:00 helios64 kernel: [ 6.715222] r8152 2-1.4:1.0 (unnamed net_device) (uninitialized): netif_napi_add_weight() called with weight 256 2023-10-19T07:01:02.167984+02:00 helios64 kernel: [ 6.730968] r8152 2-1.4:1.0 eth0: v1.12.13 2023-10-19T07:01:02.167986+02:00 helios64 kernel: [ 6.731167] usbcore: registered new interface driver r8152 In order to avoid surprises I have disabled Armbian updates for the time being (debian updates still arrive): cat /etc/apt/sources.list.d/armbian.list # deb [signed-by=/usr/share/keyrings/armbian.gpg] http://apt.armbian.com bookworm main bookworm-utils bookworm-desktop After a few days of testing I am using this now 'in production' 🙂

@prahal Thank you for the hints ! The helios64 bookworm image 23.05.4 with kernel 6.1.36 includes the mainline driver r8152 and I have tested it with both LAN interfaces 1G and 2.5G - it is stable and performs nicely. However, I could not find any matching header files for download for that kernel version: they are necessary for the dkms used on my system. Would you know a source for them (version 6.1.36) ? Unfortunately later linux versions include a regression (dwc3 usb issue) so we lose access to our USB devices i.e. using linux 6.1.50 ... Compiling current kernels and temporarily patching them to get rid of the dwc3 usb issue works, but I have the impression that I am using the wrong branch, since there are so many issues with the existing patches for rockchip64 (armbian VERSION file: 23.08.1-trunk): 74 patches to be rebased etc. Is there any better starting point than "23.08.1-trunk-6.1.xx" for building helios64 kernels ?

You could try the above and let us know if it solves the issue (should work for linux up to 6.4.10+).

Linux 6.1.36 is not affected by the dwc3 usb issue. The braking changes made it into the kernel after that. Thank you for confirming that you had no issues accessing emmc with that kernel - your system is even running from there. Your linux kernel uses the linux-tree version of the realtek driver r8152. It is based on an old version of the manufacturer and we have to test if it is reliable when used to drive the 2.5G interface. Once the dwc3 usb issue is resolved (upstream), I intend to build several versions of the linux kernel with different rtl8152 drivers so that performance and reliability can be compared.

@alchemist Thank you for the hint ! The dwc3 usb issue remains even if I compile a more recent realtek-r8152-linux driver into kernel 6.1.52. So the realtek driver does not seem to be the (only) reason for the issue. While compiling the kernel I was informed by the build system that 74 patches need to be rebased and that the rk3399-enable-dwc3-xhci-usb-trb-quirk.patch could not be applied - something has changed. This is how a system looks like that was not maintained for years ... P.S.: I will therefore currently not upgrade Helios64 (hosting all my data) to Armbian bookworm 23.05.4-6.1.36. It uses the linux-tree rtl8152 driver and needs proper testing first, in particular network performance (1Gbit/s and 2.5Gbit/s interfaces), reliability of accessing emmc and stability.

It does not matter if you refer to the current Armbian version as 23.8 as indicated on the helios64 download page or to 23.08 as indicated on the welcome screen. And it does not matter if it is 23.8.1. The important bit is that any current linux kernel or image (fully automatically) built by the Armbian build system does not provide any USB support for your Helios64 for linux kernel versions higher than 5.12. You could verify that yourself by upgrading your backup sd to the current kernel versions. Unless you have access to the original linux kernel debs you would have to reinstall your system entirely from scratch. You should therefore not upgrade your system to any current linux kernel produced by the Armbian build system until the linux rtl8152 driver is replaced for kernel versions higher than 5.12 (in this code). For the time being you could disable Armbian updates entirely (and still receive debian updates): cat /etc/apt/sources.list.d/armbian.list # deb http://apt.armbian.com bookworm main bookworm-utils bookworm-desktop

For linux versions higher than 5.12, the Armbian build system does not replace the realtek driver r8152 but it uses an aged realtek driver from the linux source tree. So if someone intends to build from sources at home, the realtek r8152 driver in the linux source tree must be replaced by a newer version, otherwise USB devices will not be accessible to helios64. It follows that all fully automated builds with the current Armbian build system building linux kernel versions higher than 5.12 will not support USB devices on helios64 anymore (until that is fixed). The good news is that there is an image available for download were the USB issue was fixed: Armbian_23.5.4_Helios64_bookworm_current_6.1.36.img (here)

A stable Armbian Bookworm configuration for your Helios64 is provided here (solved). ************************************************************************* Recently a new Armbian 23.08.1 Bookworm image with linux-6.1.50 was made available for Helios64 on its download page (see here) - which is as such great 😀. Everything starts up nicely, but unlike the previous Bookworm 23.05 image, the current one has an issue with accessing USB devices. In the boot process the following error occurs: # cat /var/log/syslog | grep error 2023-09-07T12:31:05.671598+02:00 helios64 kernel: [ 2.537009] dwc3 fe900000.usb: error -ETIMEDOUT: failed to initialize core 2023-09-07T12:31:05.671602+02:00 helios64 kernel: [ 2.537107] dwc3: probe of fe900000.usb failed with error -110 No USB device could be accessed. As this seems to be related to the realtek driver r8152, I compiled and installed the current version of that driver (see below) and after that the USB devices were accessible. # compile and install the current realtek driver git clone https://github.com/wget/realtek-r8152-linux.git cd realtek-r8152-linux... make sudo make install

That`s right - I am running bookworm off SD, since my bullseye system is still present on emmc. The effect of omitting the "BUILD_ONLY" option in the compile instruction is simply that everything is being compiled from sources such that the bookworm image is compiled too. And 'current' was used, since bookworm was designed to operate with that version of linux. A short test using 'hdparm -tT /dev/mmcblk1' #emmc indicates that emmc access speed with linux 6.1.45 is around 120MB/s (hs200), about half of what it used to be with linux 5.10.43 (220 MB/s = hs400). I can live with that, but before I copy the bookworm system from sd to emmc, could you please confirm that linux 6.1.xx can safely be used to read and write emmc using the slower hs200 mode ? P.S.: Regarding u-boot: before I became aware of your warning I already flashed the u-boot binaries to emmc using the blobs that came with the fresh bookworm image. emmc was still bootable and I did not oberve any issues. To be on the safe side, I reverted back to the original blobs present in my original bullseye image.

How to switch from network-manager to systemd-networkd is explained i.e here. Bookworm is up and running from SD for a few days now, but it would appear that emmc is still being accessed with hs200 speed.

@mrjpaxton Dist-upgrade(ing) from bullseye to bookworm did finally complete successfully. However, one should consider that device names have changed (otherwise your system may end up offline 🙂) the new interface names are: # interface names (bookworm) sd: /dev/mmcblk0 emmc: /dev/mmcblk1 eth0: end0 (1GBase-T ethernet) eth1: end1 (2.5GBase-T ethernet) P.S.: I am currently setting up bookworm from scratch starting from the fresh image to get rid of the stuff that accumulated during the last years.

@mrjpaxton I am about to upgrade linux from 5.10 to 6.1 now and if that works to dist-upgrade bullseye to bookworm. Just to prevent someone from telling us that this is not supported by Armbian - we already know that. Just in case you would like to try, here is a link to the most recent Armbian 23.08.0 - 6.1.45 bookworm image including all the linux-6.1.45 deb packages. You could also start with the fresh bookworm image, in case dist-upgrade to bookworm does not complete successfully with your installation. Armbian 23.08.0 - 6.1.45 bookworm image was compiled for Helios64 using the Armbian build system as mentioned in the parallel thread .