Helios64 - Armbian 23.08 Bookworm issues (solved)

[ebin-dev]

 

7 hours ago, mrjpaxton said:

Yes, I have also experienced a crash without the patch. It crashed when I tried to attach a HDD hard drive with a SATA-to-USB3 device for backup (I made sure the AC adapter was used for it), so it is more "stable" and responsive than before, but not completely yet. Would it be helpful to disable Armbian's ramlog and get journald to write any dumps to storage so that I can check the logs last boot, or are you guys already aware of *all* of the issues that cause these random crashes/freezes?

 

With the voltage changes proposed by @prahal (see 'opp-table 1' earlier in this thread) my system is 100% stable, absolutely reliable (using linux 6.6.x or 6.10.y). There are also others observing this.

 

Attaching drives is handled by the kernel. If that is not working, it may be that the relevant (hotplug) options are not configured.

Edited August 4 by ebin-dev

[prahal]

On 8/4/2024 at 4:03 AM, mrjpaxton said:

Yes, I have also experienced a crash without the patch. It crashed when I tried to attach a HDD hard drive with a SATA-to-USB3 device for backup (I made sure the AC adapter was used for it), so it is more "stable" and responsive than before, but not completely yet. Would it be helpful to disable Armbian's ramlog and get journald to write any dumps to storage so that I can check the logs last boot, or are you guys already aware of *all* of the issues that cause these random crashes/freezes?

 

It might be of interest in case it is not the same random corruption, then we would be able to fix the kernel. The random corruption (I believe at the CPU stage) most of the time we get a weird unrecognized instruction, but the issue still looks random (even if way more likely when btrfs scrub or zfs check).

I really need to talk to hardware guys from armbian to sort out what to take note of (USB devices, power bank, PCI stress, ...).

 

Either way you are way better with a USB c to serial cable to a computer to get the logs. You can even save the output from your serial terminal application with the "script" command, maybe in tmux/screen session.

 

[Balog Dániel]

Hello Everyone,

I have installed 6.6.39, and everything is running fine, except for one issue that I simply can't work out. 
My setup may be a bit unusual, instead of using the internal M2, I have an USB-SATA adapter plugged in to the front USB header on the board. This somehow prevents booting from SD card, and I simply can't work out what goes wrong. If I unplug the drive, all is well.

Did anyone encounter a similar issue?
 

disk.log no disk.log

[mrjpaxton]

@Balog Dániel How did you install it?

 

Since `armbian-config` is undergoing maintenance and re-haul, the best way to do it may be using `sudo nand-sata-install` at the moment, which actually seems to be part of what `armbian-config` uses anyway.

 

Follow either System on USB or SATA (Including M.2 SATA) or Transfer rootfs from eMMC to SATA or USB depending if you want to transfer or install the OS from scratch, and skip to Step 3.

 

Otherwise, if that's what you did, and that doesn't help... after reading your logs at the end, seems like the I2C driver gets stuck. I've read that it might be possible to solve the problem by disabling I2C. How to do that specifically for the Kobol, I'm not exactly sure, but I don't even have I2C drivers on my install when I check `lsmod | grep rk3x-i2c`. Not sure what it's needed for, so it should be possible to disable it with a blacklist config in `/etc/modprobe.d/` I'm guessing.

Edited August 13 by mrjpaxton

[Balog Dániel]

@mrjpaxton I flashed the SD card using etcher, (Armbian_24.5.3_Helios64_bookworm_current_6.6.36_minimal.img), apt-get update && upgrade.

 

Based on the schematics, my best guess is I2C is somehow related to the on-board battery, that's why I was hesitant to disable it (I'll take a look without the SSD drive if it's loaded).

 

I had a misadventure with flashing a corrupted disk image to MMC, that was fixed by the transfer process you have noted (the mmc currently has Armbian_21.08.2_Helios64_buster_current_5.10.63) but my main concern is to keep everything on the SD card, so I can make regular backups easily, and simply replace the card if it breaks (or revert to a backup if an upgrade somehow breaks the system, as that's what got me here in the first place). Conceptually, that should be something that should be doable right?

 

 

 

 

[mrjpaxton]

@Balog Dániel Okay, just so I am 100% clear, you say you want to boot from SD card, but why is the USB-to-SATA adapter plugged in when you are booting? Are you also trying to boot from a HDD/SSD or something? Which one would you actually prefer to boot from? Unless the SD card is above Class 10 (whatever the SDXC specifications support), it's going to be slower unlike your other 3 options: eMMC, M.2 or a SATA device over USB3.

 

You could also expose your eMMC by putting a special SD card inside. See Step 1. This will allow Offline copies since you can mount the partitions to a folder on the computer, and all you need to do is boot it with that SD card and plug it using the USB-C cable for the back.

 

But since the Kobol runs Linux, normally any Linux backup solutions will work, including `rsync`. It should be good enough to do Online backups. If you choose to use `rsync` to backup, I recommend having a script that can do `fsfreeze -f /` followed with the folders you want to rsync over like /boot, /etc, /home, /root, /var, and then `fsfreeze -u /`. It is possible, but not advisable, to convert Ext4 to Btrfs, and then you could copy over read-only snapshots as backups, which means you don't have to do the `fsfreeze` parts, but it's a pain once you have to upgrade, because Armbian still recommends upgrading with reflashes. However, it will flash an Ext4 filesystem again. I really hope they start using Btrfs sometime on some of these larger capacity devices...

 

[Balog Dániel]

@mrjpaxton

 

The setup is 5 disk RAID6 for 'cold' storage (with ~20 minute disk spin down) that I access rarely, and a 'hot' SSD to store downloads, temp files, and anything that's not OS related.
The OS should be separated from this (previously it was on MMC, but as I did not have a backup, I lost access due to something and had to reflash, thus losing all config)

Regarding the other boot option, Neither HDDs or SSD should not have any boot related files, just data.

As for the main question of why it's plugged in in the first place, I use the USB header for the front panel for an internal SSD, but as the board's SSD slot can't be used if you use all 5 disks, this was the best workaround I could find. I would really prefer to boot from SD, as I don't know the expected life of the eMMC, and how badly I may have abused it, so it's safer to expect SD card to fail and plan accordingly.

 

SD Card is UHS Class 3, and I am not doing anything that should need high IO for the OS. 

[prahal]

On 8/14/2024 at 3:52 PM, Balog Dániel said:

The setup is 5 disk RAID6 for 'cold' storage (with ~20 minute disk spin down) that I access rarely, and a 'hot' SSD to store downloads, temp files, and anything that's not OS related.
The OS should be separated from this (previously it was on MMC, but as I did not have a backup, I lost access due to something and had to reflash, thus losing all config)

Regarding the other boot option, Neither HDDs or SSD should not have any boot related files, just data.

As for the main question of why it's plugged in in the first place, I use the USB header for the front panel for an internal SSD, but as the board's SSD slot can't be used if you use all 5 disks, this was the best workaround I could find. I would really prefer to boot from SD, as I don't know the expected life of the eMMC, and how badly I may have abused it, so it's safer to expect SD card to fail and plan accordingly.

 

SD Card is UHS Class 3, and I am not doing anything that should need high IO for the OS. 

 

Thanks for the logs.

I don't know what is wrong with your boot.

 

You told that your eMMC setup broke (what you call MMC, I guess)? It was probably the eMMC breakage that affected most rk3399 boards and requires a property to be added for eMMC hs400 to boot. Since then hs400 has been disabled. I will reenable it after adding the property to the helios64 Armbian dts.

 

If I understood correctly you also would like to boot from SD (and that you are currently booting from eMMC?). But when you plug a USB external SSD (that you use to store downloads, temp files, and anything not OS related) into the front USB socket boot fails (you said "I use the USB header for the front panel for an internal SSD", I guess you mean an external USB SSD, not an internal SSD).

And you have 5 disks in the internal SATA slots as a RAID6.

 

Mind the bootloader will stay on eMMC but you can move the OS to SD. This likely won't solve your boot with a USB external SSD plugged into the front USB socket...

 

Mind I have multiple USB external HDDs plugged into the back USB socket and the boot is working.

 

What is the amperage required by the external USB drive you plug into the front USB socket? (this socket can output max 900mA).

 

ff3d0000.i2c is related to usb-c:

[    6.412784] OF: graph: no port node found in /i2c@ff3d0000/typec-portc@22

 

[Balog Dániel]

Currently I have Helios64_bullseye_current_5.10.63 on the eMMC, and if I remove the SD card, everything works, and I can boot with or without the SSD plugged in. The SD card has bookworm_6.6.36 (and upgrades), the logs are from OS on SD card with boot with and without SSD. 

The external\internal SSD is that under normal operation, I would have the SSD plugged in, within the closed case (thus 'internal' SSD), that's why it's not trivial to simply unplug it when I need to restart. 

 

I have also tested to plug in the SSD after boot (works flawlessly) and to plug this into the back USB (fails, but it just thew an error that COM is disconnected, so not really sure what the actual error is, will keep digging).
The SSD notes 1.5 A & 3.3 V, but I can't find any markings on the adapter itself (and the details on Amazon where I got it from are not specifying anything)

[prahal]

On 8/20/2024 at 6:32 PM, Balog Dániel said:

I have also tested to plug in the SSD after boot (works flawlessly) and to plug this into the back USB (fails, but it just thew an error that COM is disconnected, so not really sure what the actual error is, will keep digging).
The SSD notes 1.5 A & 3.3 V, but I can't find any markings on the adapter itself (and the details on Amazon where I got it from are not specifying anything)

 

The USB board is USB 3.0 thus should be 0.9A max and 5V.

Your SSD is 1.5*3.3 ie 4.95W, vs USB 3.0 max 4.5W. Though I believe the SSD might not always consumes the maximum. Could be later kernel use the SSD to it's maximum thus makes this SSD consumes too much. It is not given there is a bug in newer kernel.

Also it could be this extra consumption only destabilize the board at boot because the board has other components also draining more current at boot.

 

There might be tests that can help sort this out.

I think there might be ways to lower the current consumption of the M.2 SSD (maybe by lowering the libata link speed via a kernel boot param).

 

We could also try to find another USB 3 device which also stretch the limit. And check if the behavior under 5.10 and 6.6 is reproducible.

 

Can you give a link to the SSD you put on the USB M.2 board? If it is not too expensive I could try to reproduce the setup. I have a USB multimeter and could check if the USB 3 limit is really overflowed (and if the current drawn is different for 5.1 and 6.6). But I won't be able to tell soon.

 

COM error is likely your serial console program. Might be related to the helios64 board crashing on the other side. Mind that hardware hang just freeze the board, no messages are outputted. In the case I encountered it means too low voltage for load.

 

When you tell you plugged the SSD in the back USB, do you mean you plugged it after boot or before boot when it failed?

[Balog Dániel]

The manufacturer's data sheet https://www.transcend-info.com/Products/No-642

For the serial console, I am using Putty, I would assume that it's more of a board issue, but when I reconnect, i get some additional communication.

For the back USB, I plugged it in prior to power on.

 

Regarding the boot parameter, is that difficult to set up?

[prahal]

On 8/4/2024 at 3:03 AM, mrjpaxton said:

Yes, I have also experienced a crash without the patch. It crashed when I tried to attach a HDD hard drive with a SATA-to-USB3 device for backup (I made sure the AC adapter was used for it), so it is more "stable" and responsive than before, but not completely yet.

Did you mean you also experienced a crash "with" the patch when you plugged a SATA to USB3 device? (You wrote "without the patch").

Do you know the current drain of your device? ( Is the device an enclosure with a non bundled drive? Is it a 2.5 inch HDD?)

[prahal]

$ grep I2C_RK3X /boot/config-6.9.3-edge-rockchip64
CONFIG_I2C_RK3X=y

On 8/14/2024 at 1:31 AM, mrjpaxton said:

I don't even have I2C drivers on my install when I check `lsmod | grep rk3x-i2c`. Not sure what it's needed for

 

I2C is a bus to let a microprocessore communicate with other circuits.

It is enabled and cannot be disable on armbuan without rebuilding your kernel.

If I remind well it is even required on helios64 for the PMIC (the circuitry that control power, reboot, etc).

Likely you get i2c errors because the board went unstable hardware wise.

 

[prahal]

On 8/20/2024 at 10:52 PM, Balog Dániel said:

For the serial console, I am using Putty, I would assume that it's more of a board issue, but when I reconnect, i get some additional communication.

For the back USB, I plugged it in prior to power on.

 

Regarding the boot parameter, is that difficult to set up?

 

Which kind of additional communication from serial terminal?

 

The boot parameter I am thinking of might have no way to be applied to a USB SATA adapter.

This is libata.force=3.0Gbps or 1.5Gbos. But I only used it for PCIe SATA.

 

You might want ro try a dual power USB 3 cable to have enough current for the USB M.2 data drive.

 

Have you tried ebin-dev patched DTS with your USB m.2 SATA device attached (might require the edge kernel though)?

 

Edit: can you try with your USB adapter plugged to an USB 3 Power hub itself plugged to the helios64?

[magostinelli]

Hello,

following this threat, I disabled armbian update and manually installed 6.6.29 kernel with custom frequency policy.

 

I read that now Armbian officially support helios64, what do you recommend now?

 

Thanks.

[prahal]

10 hours ago, magostinelli said:

following this threat, I disabled armbian update and manually installed 6.6.29 kernel with custom frequency policy.

 

I read that now Armbian officially support helios64, what do you recommend now?

 

Don't expect much yet. If you were already on 6.6 you should not see changes.

By official support, it means I took maintainership.

I have a fix for eMMC hs400 pending but not in armbian, and a fix for the network interface MAC to fix a regression but only in 6.9 and up (armbian edge).

About the frequency policy no fix yet. Tentative hacks in the dtb voltage for the big CPUs seem to help but nothing in Armbian at this stage.

[magostinelli]

Ok, so when the new fix will be accepted, I can switch to official release, inm this way my system is updated with the new fix.

Your hacks will be included or not?

[BipBip1981]

I hope all fixes and hacks will be include soon because the result is amazing to my side 😉

 

Welcome to Armbian-unofficial 24.5.3 Bookworm with Linux 6.6.32-current-rockchip64

System load:   16%               Up time:       91 days 5:31

Edited September 8 by BipBip1981

[prahal]

On 9/6/2024 at 11:42 AM, magostinelli said:

Your hacks will be included or not?

Seeing how well the voltage hacks works on your boards I will include them in armbian (even though I still get crashes on my own board with only this 75mV hack, even though way less).

But not upstream (I am close to sending the eMMC fix upstream, I only need to read the backlog there anew to avoid too much back and forth so the patch is up to the standard). at least until I sort out why they work (I was told to try them by a board designer that told me there was a design issue with the voltage regulator which I am not up to sort out. But I checked other rk3399 armbian boards'schematics and as far as I understand they have the same design. So either all of these boards are broken and are somewhat stable for an unknown reason (maybe less stress on the big CPUs) or I misunderstood what was wrong with helios64 hardware. I need to talk to an hardware engineer.

Also I try to sort out a few other issues with other softwares and hardwares. And a few other issues. But I expect to have those in for mid October, maybe earlier.

[Flolm]

Hello,

 

I just perfom a fresh install (image Armbian_24.8.1_Helios64_bookworm_current_6.6.47_minimal.img) and copied the dtb to /boot/dtb/rockchip/rk3399-kobol-helios64.dtb.

However, I can't install Openmediavault via armbian-config (only for Buster or Bullseye). Is there any solutions to install it on Bookworm ? Or do you have an alternative to OMV ?

 

Thank for the advices

 

Flolm

EDIT: For those who are interested, I found this guide to install OMV on Armbian bookworm and it works! https://wiki.omv-extras.org/doku.php?id=omv7:armbian_bookworm_install

 

Edited October 10 by Flolm

[registr123]

HI all,

 

I have copied the 9 rtl_nic firmware files to  /lib/firmware/rtl_nic ,

However the large files transfer problem still exist.

 

v24.8.4 for Helios64 running Armbian Linux 6.6.47-current-rockchip64

 

what am I missing?

[jmanes]

@registr123 Same issue for me as of today. Sometimes SSH will cut out too if I just idle it, doesn't even require a large file transfer for me, with messages like so:

 

Bad packet length 3617372545.
ssh_dispatch_run_fatal: Connection to 10.0.0.10 port 22: Connection corrupted

 

Edited October 11 by jmanes
Wrong IP due to proxy jump

[jmanes]

Although SSH seems oddly unstable for me, I am noticing that my backups are no longer failing... so maybe that new firmware did fix something. @registr123 Did you run 'update-initramfs -u' after copying the firmware into the correct folder?

[registr123]

@jmanes Yes I did.

 

❯ dmesg | grep 2-1.4                                                                                                   
[    2.811429] usb 2-1.4: new SuperSpeed USB device number 3 using xhci-hcd
[    2.833769] usb 2-1.4: New USB device found, idVendor=0bda, idProduct=8156, bcdDevice=30.00
[    2.833809] usb 2-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=6
[    2.833826] usb 2-1.4: Product: USB 10/100/1G/2.5G LAN
[    2.833840] usb 2-1.4: Manufacturer: Realtek
[    2.833853] usb 2-1.4: SerialNumber: 000000001
[   14.509784] r8152-cfgselector 2-1.4: reset SuperSpeed USB device number 3 using xhci-hcd
[   14.843654] r8152 2-1.4:1.0: load rtl8156a-2 v2 04/27/23 successfully
[   14.943938] r8152 2-1.4:1.0 eth0: v1.12.13
[   15.037724] r8152 2-1.4:1.0 enx646266d00305: renamed from eth0
[   21.011116] r8152-cfgselector 2-1.4: reset SuperSpeed USB device number 3 using xhci-hcd
[   21.096900] r8152 2-1.4:1.0: load rtl8156a-2 v2 04/27/23 successfully
[   21.168702] r8152 2-1.4:1.0 eth0: v1.12.13
[   21.798235] r8152 2-1.4:1.0 enx646266d00305: renamed from eth0
[   28.273526] r8152 2-1.4:1.0 enx646266d00305: carrier on

 

I also lose ssh in fact all connections. Sometimes it recovers it left. Adapter restarting maybe.

 

Also noticed - if the file is below 2 to 3 GB it seams to be working fine. Files larger than 3GB trigger the problem. So yes there seams to be some improvement.

Edited October 11 by registr123

[jmanes]

@registr123 Ok we are on the same page then. I woke up this morning and saw my backup failed after around 20%, meaning it probably ran into a large file overnight. Now when it attempts to resume, it fails immediately on the same large file.  

[registr123]

I did try anything I could find as a solution but with no luck. I guess it's back to 1GB port for now. Schade

[ebin-dev]

You can further tune your system (see here).

I am on linux 6.6.56 now and use the 6.6.34 dtb with the fixes.

 

[registr123]

❯ dmesg | grep r8152
[   14.455547] usbcore: registered new device driver r8152-cfgselector
[   14.671357] r8152-cfgselector 2-1.4: reset SuperSpeed USB device number 3 using xhci-hcd
[   14.774179] r8152 2-1.4:1.0: load rtl8156a-2 v2 04/27/23 successfully
[   14.831885] r8152 2-1.4:1.0 eth0: v1.12.13
[   14.832087] usbcore: registered new interface driver r8152
[   14.906183] r8152 2-1.4:1.0 enx646266d00305: renamed from eth0
[   20.052308] r8152-cfgselector 2-1.4: reset SuperSpeed USB device number 3 using xhci-hcd
[   20.165729] r8152 2-1.4:1.0: load rtl8156a-2 v2 04/27/23 successfully
[   20.237442] r8152 2-1.4:1.0 eth0: v1.12.13
[   20.982002] r8152 2-1.4:1.0 enx646266d00305: renamed from eth0
[   45.044206] r8152 2-1.4:1.0 enx646266d00305: carrier on
[  319.888218] r8152 2-1.4:1.0 enx646266d00305: NETDEV WATCHDOG: CPU: 5: transmit queue 0 timed out 6412 ms
[  319.889163] r8152 2-1.4:1.0 enx646266d00305: Tx timeout
[  319.894396] r8152 2-1.4:1.0 enx646266d00305: Tx status -2
[  319.894464] r8152 2-1.4:1.0 enx646266d00305: Tx status -2
[  319.894525] r8152 2-1.4:1.0 enx646266d00305: Tx status -2
[  319.894582] r8152 2-1.4:1.0 enx646266d00305: Tx status -2
[  321.999005] r8152-cfgselector 2-1.4: reset SuperSpeed USB device number 3 using xhci-hcd
[ 1370.733798] r8152 2-1.4:1.0 enx646266d00305: NETDEV WATCHDOG: CPU: 5: transmit queue 0 timed out 6380 ms
[ 1370.734704] r8152 2-1.4:1.0 enx646266d00305: Tx timeout
[ 1370.737585] r8152 2-1.4:1.0 enx646266d00305: Tx status -2
[ 1370.737684] r8152 2-1.4:1.0 enx646266d00305: Tx status -2
[ 1370.737755] r8152 2-1.4:1.0 enx646266d00305: Tx status -2
[ 1370.737816] r8152 2-1.4:1.0 enx646266d00305: Tx status -2
[ 1372.846993] r8152-cfgselector 2-1.4: reset SuperSpeed USB device number 3 using xhci-hcd

 

@ebin-dev  Thanks. I have implemented the tweaks you pointed to. For me the problem persists. Maybe I have not done everything correctly. I will start from scratch. 

 

it seams that the the tweaks doubled the "size" - now I can transfer files up to 5G without the adapter hanging. Kinda usable, but still I will probably have to move back to the gigabit adapter. 

Edited October 12 by registr123

[jmanes]

I am unfortunately seeing this instability even with the 1GB ethernet port... Not sure why. 

EDIT: Followed tweaks posted earlier, 1GB is now stable. 

Edited October 15 by jmanes

[BipBip1981]

Hi evevrybody,

 

Since few days i note a problem with SATA on my Helios64.

 

Only on a cold boot, SATA speed isn't to 6.0Gbps randomly on SATA disk, after reboot command, all it's good at 6.0Gbps

 

root@helios64:~# dmesg | grep SATA
[    3.169958] ahci 0000:01:00.0: AHCI 0001.0301 32 slots 5 ports 6 Gbps 0x1f impl SATA mode
[    3.183007] ata1: SATA max UDMA/133 abar m8192@0xfa010000 port 0xfa010100 irq 74
[    3.183681] ata2: SATA max UDMA/133 abar m8192@0xfa010000 port 0xfa010180 irq 75
[    3.184378] ata3: SATA max UDMA/133 abar m8192@0xfa010000 port 0xfa010200 irq 76
[    3.185046] ata4: SATA max UDMA/133 abar m8192@0xfa010000 port 0xfa010280 irq 77
[    3.185710] ata5: SATA max UDMA/133 abar m8192@0xfa010000 port 0xfa010300 irq 78
[   10.440028] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[   10.976033] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[   11.480029] ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[   11.963976] ata4: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[   12.472056] ata5: SATA link up 6.0 Gbps (SStatus 133 SControl 300)

 

After reboot command:

 

root@helios64:~# dmesg | grep SATA
[    3.151099] ahci 0000:01:00.0: AHCI 0001.0301 32 slots 5 ports 6 Gbps 0x1f impl SATA mode
[    3.164109] ata1: SATA max UDMA/133 abar m8192@0xfa010000 port 0xfa010100 irq 74
[    3.164781] ata2: SATA max UDMA/133 abar m8192@0xfa010000 port 0xfa010180 irq 75
[    3.165448] ata3: SATA max UDMA/133 abar m8192@0xfa010000 port 0xfa010200 irq 76
[    3.166114] ata4: SATA max UDMA/133 abar m8192@0xfa010000 port 0xfa010280 irq 77
[    3.166778] ata5: SATA max UDMA/133 abar m8192@0xfa010000 port 0xfa010300 irq 78
[   10.000101] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[   10.524107] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[   11.036103] ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[   11.520039] ata4: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[   12.024104] ata5: SATA link up 6.0 Gbps (SStatus 133 SControl 300)

 

I upgrade to 24.11 trunk build myself with framework and tune with Prahal dtb file and 6.6.56 kernel.

 

Keep in touch.

 

Bye

Edited October 16 by BipBip1981