prahal

Awesome. And thanks a lot for the feedback. Could you explain which side you disconnect/reconnected? The HDD side only? Or on the motherboard too? I believe HDD side is enough, just want a clue if that could be wrong. I believe the connector were not clean, or a bit oxidized out of fabric (maybe connectors were stored in an area with an aggressive climate... I am no expert, but I clue that with the parts being serviced during Covid mess, some unusual process happened). It is not like the hardware is bad, only "dirty". Here I had bad lost connectivity to an HDD, extracting and inserting it a few times back seemed to have cleaned the connectors (I also put isopropanol on them, but I don't remind if I brushed them at that time).

-19 in ata1: link is slow to respond, please be patient (ready=-19) is ENODEV in https://github.com/torvalds/linux/blob/6efbea77b390604a7be7364583e19cd2d6a1291b/drivers/ata/libata-core.c#L3594 but ready is resetted to 0 else the function would exit before this message https://github.com/torvalds/linux/blob/6efbea77b390604a7be7364583e19cd2d6a1291b/drivers/ata/libata-core.c#L3577 As I told, I had bad contact on my sata ports, try removing and inserting the HDD in the sata socket a few times. That might remove things on the contacts. On my side I also clean the sata sockets with isopropanol when I bought some (I use 99,5% isopropanol but I don't know if less concentrated is OK). You might paste the complete log around the ata1 lines in the kernel log. If the issue is always with the same sata port, you could try swapping the HDD to see if the issue follows the HDD or if this is the link or port. But indeed I guess the issue is hardware.

@BipBip1981 Could you grep for "ata", or check the logs for ata errors and paste them? Or tell if there are no other messages with ata in the logs? Don't you have any "hard resetting link" messages in the kernel logs? On my side I once had drives that were not detected extracting the drive from the SATA power and data socket a few times (and cleaning them with isopropanol once, might have helped) did it. I believe my socket were oxidized though that is a wild guess. Issue gone either way.

Seeing how well the voltage hacks works on your boards I will include them in armbian (even though I still get crashes on my own board with only this 75mV hack, even though way less). But not upstream (I am close to sending the eMMC fix upstream, I only need to read the backlog there anew to avoid too much back and forth so the patch is up to the standard). at least until I sort out why they work (I was told to try them by a board designer that told me there was a design issue with the voltage regulator which I am not up to sort out. But I checked other rk3399 armbian boards'schematics and as far as I understand they have the same design. So either all of these boards are broken and are somewhat stable for an unknown reason (maybe less stress on the big CPUs) or I misunderstood what was wrong with helios64 hardware. I need to talk to an hardware engineer. Also I try to sort out a few other issues with other softwares and hardwares. And a few other issues. But I expect to have those in for mid October, maybe earlier.

Don't expect much yet. If you were already on 6.6 you should not see changes. By official support, it means I took maintainership. I have a fix for eMMC hs400 pending but not in armbian, and a fix for the network interface MAC to fix a regression but only in 6.9 and up (armbian edge). About the frequency policy no fix yet. Tentative hacks in the dtb voltage for the big CPUs seem to help but nothing in Armbian at this stage.

Hi. I am interested as I want to ease maintaining Helios64 by getting a second board. Would you be shipping to France?

I guess this is one core not responding anymore, likely CPU 5 (one of the big cores). Which kernel do you run? Is this the first time you encounter this bug? You might want to run ebin-dev dtb (there are voltage hacks for the big CPUs in it).

Which kind of additional communication from serial terminal? The boot parameter I am thinking of might have no way to be applied to a USB SATA adapter. This is libata.force=3.0Gbps or 1.5Gbos. But I only used it for PCIe SATA. You might want ro try a dual power USB 3 cable to have enough current for the USB M.2 data drive. Have you tried ebin-dev patched DTS with your USB m.2 SATA device attached (might require the edge kernel though)? Edit: can you try with your USB adapter plugged to an USB 3 Power hub itself plugged to the helios64?

$ grep I2C_RK3X /boot/config-6.9.3-edge-rockchip64 CONFIG_I2C_RK3X=y I2C is a bus to let a microprocessore communicate with other circuits. It is enabled and cannot be disable on armbuan without rebuilding your kernel. If I remind well it is even required on helios64 for the PMIC (the circuitry that control power, reboot, etc). Likely you get i2c errors because the board went unstable hardware wise.

Did you mean you also experienced a crash "with" the patch when you plugged a SATA to USB3 device? (You wrote "without the patch"). Do you know the current drain of your device? ( Is the device an enclosure with a non bundled drive? Is it a 2.5 inch HDD?)

The USB board is USB 3.0 thus should be 0.9A max and 5V. Your SSD is 1.5*3.3 ie 4.95W, vs USB 3.0 max 4.5W. Though I believe the SSD might not always consumes the maximum. Could be later kernel use the SSD to it's maximum thus makes this SSD consumes too much. It is not given there is a bug in newer kernel. Also it could be this extra consumption only destabilize the board at boot because the board has other components also draining more current at boot. There might be tests that can help sort this out. I think there might be ways to lower the current consumption of the M.2 SSD (maybe by lowering the libata link speed via a kernel boot param). We could also try to find another USB 3 device which also stretch the limit. And check if the behavior under 5.10 and 6.6 is reproducible. Can you give a link to the SSD you put on the USB M.2 board? If it is not too expensive I could try to reproduce the setup. I have a USB multimeter and could check if the USB 3 limit is really overflowed (and if the current drawn is different for 5.1 and 6.6). But I won't be able to tell soon. COM error is likely your serial console program. Might be related to the helios64 board crashing on the other side. Mind that hardware hang just freeze the board, no messages are outputted. In the case I encountered it means too low voltage for load. When you tell you plugged the SSD in the back USB, do you mean you plugged it after boot or before boot when it failed?

My change was about restoring the eth0/end0 MAC address to its intended value (ie grabbed from OTP via SPI as it was designed to work until it was broken at one point). It does not change the interface names. I warned about it as if one had a static DHCP lease on the MAC that was set for a few years other than the OTP one the lease would not apply. The change is included for 6.9 and up, the issue reported above is for 6.8 (though it is likely a user space issue as interface renaming is udev userspace). So I guess @crosser upgraded from a kernel with the initial behavior I restored to an intermediate one which did not grab the MAC from the OTP. Note also that I did not apply these changes to the 6.6 kernel (-current), only to -edge. The MAC address and the interface rename issue are unrelated.

Thanks for the logs. I don't know what is wrong with your boot. You told that your eMMC setup broke (what you call MMC, I guess)? It was probably the eMMC breakage that affected most rk3399 boards and requires a property to be added for eMMC hs400 to boot. Since then hs400 has been disabled. I will reenable it after adding the property to the helios64 Armbian dts. If I understood correctly you also would like to boot from SD (and that you are currently booting from eMMC?). But when you plug a USB external SSD (that you use to store downloads, temp files, and anything not OS related) into the front USB socket boot fails (you said "I use the USB header for the front panel for an internal SSD", I guess you mean an external USB SSD, not an internal SSD). And you have 5 disks in the internal SATA slots as a RAID6. Mind the bootloader will stay on eMMC but you can move the OS to SD. This likely won't solve your boot with a USB external SSD plugged into the front USB socket... Mind I have multiple USB external HDDs plugged into the back USB socket and the boot is working. What is the amperage required by the external USB drive you plug into the front USB socket? (this socket can output max 900mA). ff3d0000.i2c is related to usb-c: [ 6.412784] OF: graph: no port node found in /i2c@ff3d0000/typec-portc@22

It might be of interest in case it is not the same random corruption, then we would be able to fix the kernel. The random corruption (I believe at the CPU stage) most of the time we get a weird unrecognized instruction, but the issue still looks random (even if way more likely when btrfs scrub or zfs check). I really need to talk to hardware guys from armbian to sort out what to take note of (USB devices, power bank, PCI stress, ...). Either way you are way better with a USB c to serial cable to a computer to get the logs. You can even save the output from your serial terminal application with the "script" command, maybe in tmux/screen session.

mind the voltage changes I am uneasy to push to main until I either get no more crashes for a long long time or the issue is nailed down to its cause. I might revisit this choice. I plan to have the emmc hs400 fix in Armbian and vanilla linux. Probably not for the august Armbian release but I expect for the next one.