prahal

I won't be able to test newer image until at least a week, away from the board. Before leaving, I was able to confirm that I get back panel sound with rock-5-itx_debian_bullseye_kde_b6.img.xz image (kernel 5.10.110-37-rockchip). You might want to provide amixer output to sort out any volume level issue. `cat /proc/asound/cards` `amixer` and `amixer -c 3` if card 3 is `rockchipes8316` in `cat /proc/asound/cards` output.

Yes I planned them for 24.11 but had personal issue and also I took part in another board mainternship. The emmc patch need only that I sort out of to apply it to the current patfhset. I asked for insight on how to handle reformatting existing patch that I modify and was told that I should submit any changes to a patch in a separate patch by Werner but I am still uneasy to small patch files instead of fixing existing patches. I wonder if I explain my situation clearly ... The voltage patch needs a little more work as I made many instances of the voltage change on my side, so I need to disassemble ebin-dev dtb though it is achievable in a few hours. In short I tried learn ingambian processes first, had a little more work on another board (mostly learning required additional hardware and ordering it, then handlingna few glitches like power supply dying) then personal matters and taking part in gatherings. I now plan these commits for 25.02.

I also think of a hardware issue. If you have a voltmeter you could check if you have 12V in the board (taking the ground on metal case around connector (ethernet, etc). Might be a repair shop could fix it. I don't know the cost of such a repair.

I don't know why you don't get a shell on the serial console. But we can get the network issue worked around. There are two ethernet interfaces: end0 the 1Gbls and the second in your cawe enx646266d0034d the 2.5Gbps ethernet one. The enx646266d0034d is MAC hw address 64:62:66:d0:03:4d (I've the mac addr is in the interface name). The end0 should be the same minus one is 64:62:66:d0:03:4c. What are the status of the LEDs on the front panel (blinking/solid)?

@crosser was it stable with the other dtb?

I believe it is supported https://wiki.kobol.io/helios64/hardware/ in the comments

You could check with a voltmeter on the board for the 12V. Schematics are here https://wiki.kobol.io/helios64/docs/ . Do you have serial console output ? Though I believe with such an issue this is highly unlikely. From https://wiki.kobol.io/helios64/led/ comments, the system status LED and LAN LED are software controlled and HDD Activity LEDs hardware controlled. Though highly liekly LED1 (system rail power) is hardware controlled.

Awesome. And thanks a lot for the feedback. Could you explain which side you disconnect/reconnected? The HDD side only? Or on the motherboard too? I believe HDD side is enough, just want a clue if that could be wrong. I believe the connector were not clean, or a bit oxidized out of fabric (maybe connectors were stored in an area with an aggressive climate... I am no expert, but I clue that with the parts being serviced during Covid mess, some unusual process happened). It is not like the hardware is bad, only "dirty". Here I had bad lost connectivity to an HDD, extracting and inserting it a few times back seemed to have cleaned the connectors (I also put isopropanol on them, but I don't remind if I brushed them at that time).

-19 in ata1: link is slow to respond, please be patient (ready=-19) is ENODEV in https://github.com/torvalds/linux/blob/6efbea77b390604a7be7364583e19cd2d6a1291b/drivers/ata/libata-core.c#L3594 but ready is resetted to 0 else the function would exit before this message https://github.com/torvalds/linux/blob/6efbea77b390604a7be7364583e19cd2d6a1291b/drivers/ata/libata-core.c#L3577 As I told, I had bad contact on my sata ports, try removing and inserting the HDD in the sata socket a few times. That might remove things on the contacts. On my side I also clean the sata sockets with isopropanol when I bought some (I use 99,5% isopropanol but I don't know if less concentrated is OK). You might paste the complete log around the ata1 lines in the kernel log. If the issue is always with the same sata port, you could try swapping the HDD to see if the issue follows the HDD or if this is the link or port. But indeed I guess the issue is hardware.

@BipBip1981 Could you grep for "ata", or check the logs for ata errors and paste them? Or tell if there are no other messages with ata in the logs? Don't you have any "hard resetting link" messages in the kernel logs? On my side I once had drives that were not detected extracting the drive from the SATA power and data socket a few times (and cleaning them with isopropanol once, might have helped) did it. I believe my socket were oxidized though that is a wild guess. Issue gone either way.

Seeing how well the voltage hacks works on your boards I will include them in armbian (even though I still get crashes on my own board with only this 75mV hack, even though way less). But not upstream (I am close to sending the eMMC fix upstream, I only need to read the backlog there anew to avoid too much back and forth so the patch is up to the standard). at least until I sort out why they work (I was told to try them by a board designer that told me there was a design issue with the voltage regulator which I am not up to sort out. But I checked other rk3399 armbian boards'schematics and as far as I understand they have the same design. So either all of these boards are broken and are somewhat stable for an unknown reason (maybe less stress on the big CPUs) or I misunderstood what was wrong with helios64 hardware. I need to talk to an hardware engineer. Also I try to sort out a few other issues with other softwares and hardwares. And a few other issues. But I expect to have those in for mid October, maybe earlier.

Don't expect much yet. If you were already on 6.6 you should not see changes. By official support, it means I took maintainership. I have a fix for eMMC hs400 pending but not in armbian, and a fix for the network interface MAC to fix a regression but only in 6.9 and up (armbian edge). About the frequency policy no fix yet. Tentative hacks in the dtb voltage for the big CPUs seem to help but nothing in Armbian at this stage.

Hi. I am interested as I want to ease maintaining Helios64 by getting a second board. Would you be shipping to France?

I guess this is one core not responding anymore, likely CPU 5 (one of the big cores). Which kernel do you run? Is this the first time you encounter this bug? You might want to run ebin-dev dtb (there are voltage hacks for the big CPUs in it).

Which kind of additional communication from serial terminal? The boot parameter I am thinking of might have no way to be applied to a USB SATA adapter. This is libata.force=3.0Gbps or 1.5Gbos. But I only used it for PCIe SATA. You might want ro try a dual power USB 3 cable to have enough current for the USB M.2 data drive. Have you tried ebin-dev patched DTS with your USB m.2 SATA device attached (might require the edge kernel though)? Edit: can you try with your USB adapter plugged to an USB 3 Power hub itself plugged to the helios64?

$ grep I2C_RK3X /boot/config-6.9.3-edge-rockchip64 CONFIG_I2C_RK3X=y I2C is a bus to let a microprocessore communicate with other circuits. It is enabled and cannot be disable on armbuan without rebuilding your kernel. If I remind well it is even required on helios64 for the PMIC (the circuitry that control power, reboot, etc). Likely you get i2c errors because the board went unstable hardware wise.

Did you mean you also experienced a crash "with" the patch when you plugged a SATA to USB3 device? (You wrote "without the patch"). Do you know the current drain of your device? ( Is the device an enclosure with a non bundled drive? Is it a 2.5 inch HDD?)

The USB board is USB 3.0 thus should be 0.9A max and 5V. Your SSD is 1.5*3.3 ie 4.95W, vs USB 3.0 max 4.5W. Though I believe the SSD might not always consumes the maximum. Could be later kernel use the SSD to it's maximum thus makes this SSD consumes too much. It is not given there is a bug in newer kernel. Also it could be this extra consumption only destabilize the board at boot because the board has other components also draining more current at boot. There might be tests that can help sort this out. I think there might be ways to lower the current consumption of the M.2 SSD (maybe by lowering the libata link speed via a kernel boot param). We could also try to find another USB 3 device which also stretch the limit. And check if the behavior under 5.10 and 6.6 is reproducible. Can you give a link to the SSD you put on the USB M.2 board? If it is not too expensive I could try to reproduce the setup. I have a USB multimeter and could check if the USB 3 limit is really overflowed (and if the current drawn is different for 5.1 and 6.6). But I won't be able to tell soon. COM error is likely your serial console program. Might be related to the helios64 board crashing on the other side. Mind that hardware hang just freeze the board, no messages are outputted. In the case I encountered it means too low voltage for load. When you tell you plugged the SSD in the back USB, do you mean you plugged it after boot or before boot when it failed?

My change was about restoring the eth0/end0 MAC address to its intended value (ie grabbed from OTP via SPI as it was designed to work until it was broken at one point). It does not change the interface names. I warned about it as if one had a static DHCP lease on the MAC that was set for a few years other than the OTP one the lease would not apply. The change is included for 6.9 and up, the issue reported above is for 6.8 (though it is likely a user space issue as interface renaming is udev userspace). So I guess @crosser upgraded from a kernel with the initial behavior I restored to an intermediate one which did not grab the MAC from the OTP. Note also that I did not apply these changes to the 6.6 kernel (-current), only to -edge. The MAC address and the interface rename issue are unrelated.

Thanks for the logs. I don't know what is wrong with your boot. You told that your eMMC setup broke (what you call MMC, I guess)? It was probably the eMMC breakage that affected most rk3399 boards and requires a property to be added for eMMC hs400 to boot. Since then hs400 has been disabled. I will reenable it after adding the property to the helios64 Armbian dts. If I understood correctly you also would like to boot from SD (and that you are currently booting from eMMC?). But when you plug a USB external SSD (that you use to store downloads, temp files, and anything not OS related) into the front USB socket boot fails (you said "I use the USB header for the front panel for an internal SSD", I guess you mean an external USB SSD, not an internal SSD). And you have 5 disks in the internal SATA slots as a RAID6. Mind the bootloader will stay on eMMC but you can move the OS to SD. This likely won't solve your boot with a USB external SSD plugged into the front USB socket... Mind I have multiple USB external HDDs plugged into the back USB socket and the boot is working. What is the amperage required by the external USB drive you plug into the front USB socket? (this socket can output max 900mA). ff3d0000.i2c is related to usb-c: [ 6.412784] OF: graph: no port node found in /i2c@ff3d0000/typec-portc@22

It might be of interest in case it is not the same random corruption, then we would be able to fix the kernel. The random corruption (I believe at the CPU stage) most of the time we get a weird unrecognized instruction, but the issue still looks random (even if way more likely when btrfs scrub or zfs check). I really need to talk to hardware guys from armbian to sort out what to take note of (USB devices, power bank, PCI stress, ...). Either way you are way better with a USB c to serial cable to a computer to get the logs. You can even save the output from your serial terminal application with the "script" command, maybe in tmux/screen session.

mind the voltage changes I am uneasy to push to main until I either get no more crashes for a long long time or the issue is nailed down to its cause. I might revisit this choice. I plan to have the emmc hs400 fix in Armbian and vanilla linux. Probably not for the august Armbian release but I expect for the next one.

Adding more clues from your issue: this u-boot is I guess, mainlain u-boot and ATF with rockchip DDR blob. linux-u-boot-edge-helios64_22.02.1 is likely the rockchip ATF, miniloader and DDR blob. Seems linux-u-boot-edge-helios64_22.02.1_arm64 is not available to download anymore? EIther way, this is a binary blob so could only be used to test feature parity. apt policy linux-u-boot-helios64-edge linux-u-boot-helios64-edge: Installé : 24.5.1 Candidat : 24.5.1 Table de version : *** 24.5.1 500 500 http://apt.armbian.com bookworm/main arm64 Packages 100 /var/lib/dpkg/status 24.2.1 500 500 http://apt.armbian.com bookworm/main arm64 Packages the link you provided looks like a match for this issue. I noted that rockchip-i2s ff8a0000.i2s: Could not register PCM is printed each time a USB device is plugged in (probably also when probed at boot). also, it has been a long time I noticed: juin 08 16:18:53 helios64 kernel: platform ff1e0000.spi: deferred probe pending: (reason unknown) juin 08 16:18:53 helios64 kernel: platform ff200000.spi: deferred probe pending: (reason unknown) juin 08 16:18:53 helios64 kernel: platform ff8a0000.i2s: deferred probe pending: (reason unknown) juin 08 16:18:53 helios64 kernel: amba ff6d0000.dma-controller: deferred probe pending: (reason unknown) juin 08 16:18:53 helios64 kernel: amba ff6e0000.dma-controller: deferred probe pending: (reason unknown) juin 08 16:18:53 helios64 kernel: platform ff1d0000.spi: deferred probe pending: (reason unknown) at the end of the kernel boot log. (ie kernel giving up trying to load the drivers for these). Seems related too.

Linux kernel ML discussion about an upstream fix WIP as of the 11th of June 2024: https://lore.kernel.org/lkml/20240326-rk-default-enable-strobe-pulldown-v1-3-f410c71605c0@folker-schwesinger.de/t/