prahal

@BinaryWaves see my post above for how to restore the files u-boot needs to start the kernel I have not yet been able to find out the cause of the random (sometimes often, sometimes months apart, crashes of the board). However, such crashes tend to corrupt files more easily on SD and EMMC than on HDD probably due to the block size. So you can end up unable to boot after such crashes due to file corruption. I hope one day to get rid of at least one such crash cause (if more than one cause is at stake) but not there yet. So one has to resort to chroot to the SD or EMMC and reinstall. Or reinstall a new image if one does not want to bother learning the required steps for chroot. You could also take OS FS backup images regularly until the issue is sorted out and reimage after a crash that broke boot (though they are pretty rare, so one might end up stopping imaging until it happens).

I can no longer reproduce this rescan behavior (LEDs stay solid blue before triggering on disk accesses). It could have been fixed days ago. I am currently on 6.6.7-edge-rockchip64 fetch and build on Armbian three days ago. U-Boot 2022.07-armbian (Jul 21 2023 - 02:01:45 +0000)

6.6 might be unstable, you could try the armbian repo 6.2 , that is linux-image-edge-rockchip64, linux-headers-edge-rockchip64 and linux-dtb-edge-rockchip64 at version 23.11.1 (mind if you have both current and edge kernel there are issues when a current upgrade is push while you have a edge version previously installed, this is an armbian packaging issue though. BUt for the matter of testing it is fine). Then ssh to the helios64 and edit the current dts with "armbian-config > System > Dtc" emmc_phy settings are in the "phy@f780" node (the default editor is nano, you can search with Ctrl+W). Add " rockchip,enable-strobe-pulldown; " under this node then under the mmc@fe330000 node add: mmc-hs400-1_8v; mmc-hs400-enhanced-strobe; confirm and reboot. It could be that the pulldown setting is enough even before linux 6.1 (there were other core regulator fixes, but the most important one was added in 6.1), so you could try with lower kernel version. root@helios64:~# hdparm -tT /dev/mmcblk0 /dev/mmcblk0: Timing cached reads: 1888 MB in 2.00 seconds = 944.72 MB/sec Timing buffered disk reads: 552 MB in 3.00 seconds = 183.86 MB/sec Note that EMMC on 6.6.7 /dev/mmcblk0 and SD /dev/mmcblk1 hdparm -tT /dev/mmcblk1 /dev/mmcblk1: Timing cached reads: 1858 MB in 2.00 seconds = 930.17 MB/sec Timing buffered disk reads: 128 MB in 3.10 seconds = 41.34 MB/sec

@piter75@ebin-dev Feel free to confirm the rockchip emmc phy pulldown dts indeed fixes emmc write in hs400es mode (I believe on most rk3399 boards, not only helios64). Likely on kernel up and above 6.1 (for the regulator fixes). Thanks to @RussianNeuroMancer I found: arm64: dts: rockchip: add enable-strobe-pulldown to emmc phy on nanopi4 https://github.com/torvalds/linux/commit/463be3cb357dab7d7e4d8dcc7c15c642e10c5bef and: phy: rockchip: set pulldown for strobe line in dts https://github.com/torvalds/linux/commit/8b5c2b45b8f0a11c9072da0f7baf9ee986d3151e which might be related to this emmc issue (as noted in the mainline patch in the rockchip kernel this emmc pulldown setting is always enabled while in mainline it is disabled by default). This patch was introduced in 5.11 which might explain why even with the regulator issue fixed (the one I found by regression testing) with kernel above 5.11 I still had hs400 failing on me. With this emmc_phy pulldown setting in the dts I can now write to the emmc in hs400es mode ! means one have to add "rockchip,enable-strobe-pulldown;" in the emmc_phy node for most boards --- arch/arm64/boot/dts/rockchip/rk3399-kobol-helios64.dts | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/arch/arm64/boot/dts/rockchip/rk3399-kobol-helios64.dts b/arch/arm64/boot/dts/rockchip/rk3399-kobol-helios64.dts index 69d76dea35d0..3c1965660fbd 100644 --- a/arch/arm64/boot/dts/rockchip/rk3399-kobol-helios64.dts +++ b/arch/arm64/boot/dts/rockchip/rk3399-kobol-helios64.dts @@ -436,10 +436,11 @@ &cpu_b0 { &cpu_b1 { cpu-supply = <&vdd_cpu_b>; }; &emmc_phy { + rockchip,enable-strobe-pulldown; status = "okay"; }; &gmac { assigned-clocks = <&cru SCLK_RMII_SRC>; @@ -965,13 +966,12 @@ &saradc { &sdhci { assigned-clock-rates = <150000000>; bus-width = <8>; mmc-hs200-1_8v; - // hs400 is broken on Helios64 since 5.10.60 - // mmc-hs400-1_8v; - // mmc-hs400-enhanced-strobe; + mmc-hs400-1_8v; + mmc-hs400-enhanced-strobe; supports-emmc; non-removable; disable-wp; status = "okay"; vqmmc-supply = <&vcc1v8_sys_s0>; -- There is also the regulator fix. Back then I bisected the first bad commit 06653ebc0ad2e0b7d799cd71a5c2933ed2fb7a66 which seems to have been a backport of the master branch to commit https://github.com/torvalds/linux/commit/98e48cd9283dbac0e1445ee780889f10b3d1db6a "regulator: core: resolve supply for boot-on/always-on regulators" introduced in 5.13. I am pretty confident the regulator issues were fixed at least in linux 6.1. Note that the emmc phy pulldown code was not backported to 5.10.y (https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/log/drivers/phy/rockchip/phy-rockchip-emmc.c?h=linux-5.10.y) So I believe it was first included in 5.11. I no longer have the following CQE errors when writing to emmc in hs400es mode: [ 18.985162] mmc1: running CQE recovery [ 18.988056] ------------[ cut here ]------------ [ 18.988500] mmc1: cqhci: spurious TCN for tag 12 [ 18.989019] WARNING: CPU: 0 PID: 269 at drivers/mmc/host/cqhci-core.c:787 cqhci_irq+0x4b4/0x640 [ 18.989838] Modules linked in: quota_v2 quota_tree r8152 snd_soc_hdmi_codec ftdi_sio usbserial snd_soc_rockchip_i2s leds_pwm snd_soc_rockchip_pcm snd_soc_core snd_pcm_dmaengine snd_pcm pwm_fan gpio_charger snd_timer panfrost gpu_sched snd hantro_vpu(C) soundcore rockchip_vdec(C) v4l2_h264 rockchip_rga rockchip_iep videobuf2_dma_contig videobuf2_vmalloc videobuf2_dma_sg v4l2_mem2mem sg videobuf2_memops videobuf2_v4l2 videobuf2_common fusb302 videodev tcpm typec mc gpio_beeper cpufreq_dt nfsd dm_mod auth_rpcgss nfs_acl lockd grace ledtrig_netdev sunrpc lm75 ip_tables x_tables autofs4 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx raid1 raid0 multipath linear raid10 uas realtek md_mod dwmac_rk stmmac_platform stmmac pcs_xpcs adc_keys [ 18.995876] CPU: 0 PID: 269 Comm: kworker/0:1H Tainted: G C 5.15.29-rockchip64 #trunk.0010 [ 18.996740] Hardware name: Helios64 (DT) [ 18.997104] Workqueue: kblockd blk_mq_run_work_fn [ 18.997555] pstate: 800000c5 (Nzcv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 18.998182] pc : cqhci_irq+0x4b4/0x640 [ 18.998537] lr : cqhci_irq+0x4b4/0x640 [ 18.998880] sp : ffff800008003d10 [ 18.999179] x29: ffff800008003d10 x28: ffff0000012b8e80 x27: ffff000004c2d580 [ 18.999819] x26: ffff00000652fc98 x25: ffff8000094d0888 x24: ffff800009c975e8 [ 19.000462] x23: ffff8000094f49c8 x22: 0000000000000002 x21: ffff000004c2d000 [ 19.001107] x20: 000000000000000c x19: ffff00000652fc80 x18: 0000000000000010 [ 19.001754] x17: ffff8000ee03c000 x16: ffff800008004000 x15: 00000000000002f6 [ 19.002400] x14: ffff800008003a20 x13: 00000000ffffffea x12: ffff800009abfbc8 [ 19.003047] x11: 0000000000000003 x10: ffff800009aa7b88 x9 : ffff800009aa7be0 [ 19.003694] x8 : 0000000000017fe8 x7 : c0000000ffffefff x6 : 0000000000000001 [ 19.004338] x5 : ffff8000ee03c000 x4 : 0000000000000000 x3 : 0000000000010004 [ 19.004980] x2 : 0000000000010003 x1 : f94ce7e57b395a00 x0 : 0000000000000000 [ 19.005623] Call trace: [ 19.005852] cqhci_irq+0x4b4/0x640 [ 19.006169] sdhci_arasan_cqhci_irq+0x5c/0x88 [ 19.006564] sdhci_irq+0xcc/0x10c0 [ 19.006876] __handle_irq_event_percpu+0x60/0x250 [ 19.007302] handle_irq_event_percpu+0x38/0x90 [ 19.007705] handle_irq_event+0x48/0xe8 [ 19.008058] handle_fasteoi_irq+0xb8/0x148 [ 19.008430] handle_domain_irq+0x90/0xd8 [ 19.008788] gic_handle_irq+0xb8/0x138 [ 19.009131] call_on_irq_stack+0x28/0x50 [ 19.009492] do_interrupt_handler+0x58/0x68 [ 19.009874] el1_interrupt+0x30/0x78 [ 19.010202] el1h_64_irq_handler+0x18/0x28 [ 19.010575] el1h_64_irq+0x74/0x78 [ 19.010889] preempt_count_sub+0x34/0xc0 [ 19.011251] _raw_spin_unlock_irqrestore+0x20/0x40 [ 19.011686] sdhci_cqe_enable+0x130/0x228 [ 19.012060] sdhci_arasan_cqe_enable+0x94/0xb8 [ 19.012465] cqhci_request+0xd0/0x650 [ 19.012808] mmc_cqe_start_req+0xb4/0x198 [ 19.013177] mmc_blk_mq_issue_rq+0x494/0x9a8 [ 19.013567] mmc_mq_queue_rq+0x114/0x2b0 [ 19.013928] blk_mq_dispatch_rq_list+0x120/0x7e8 [ 19.014355] __blk_mq_sched_dispatch_requests+0xc4/0x1e0 [ 19.014835] blk_mq_sched_dispatch_requests+0x3c/0x78 [ 19.015291] __blk_mq_run_hw_queue+0x64/0xa0 [ 19.015678] blk_mq_run_work_fn+0x20/0x30 [ 19.016042] process_one_work+0x20c/0x4c8 [ 19.016407] worker_thread+0x48/0x478 [ 19.016739] kthread+0x138/0x150 [ 19.017033] ret_from_fork+0x10/0x20 [ 19.017358] ---[ end trace 70e1eee6af816777 ]--- [ 19.020419] mmc1: running CQE recovery [ 19.025922] mmc1: running CQE recovery [ 19.028135] blk_update_request: I/O error, dev mmcblk1, sector 1605640 op 0x1:(WRITE) flags 0x800 phys_seg 57 prio class 0 [ 19.029190] Buffer I/O error on dev mmcblk1p1, logical block 196609, lost sync page write [ 19.029989] EXT4-fs error (device mmcblk1p1): ext4_check_bdev_write_error:218: comm sed: Error while async write back metadata [ 19.030191] Aborting journal on device mmcblk1p1-8. [ 19.032803] EXT4-fs (mmcblk1p1): Remounting filesystem read-only [ 19.033426] EXT4-fs error (device mmcblk1p1) in __ext4_new_inode:1136: Journal has aborted Feel free to submit this patch to Armbian github or even to the mainline helios64 dts on your side.

@ebin-dev note that u-boot is not updated in the device when the u-boot package is updated. I really should have pushed the workaround since I cooked it up, but apt upgrade cannot break the u-boot on the board. I will try to send the pull request for this u-boot workaround in January. Note also that it is only a workaround. The support for LPDDR4 on u-boot mainline might be buggy. But I don't know how I can fix it on my side. The rockchip blob that works is a binary without sources. Maybe one could ask rockchip devs to cook a fix for u-boot mainline?

What do you mean? hs400 is still there. I did not check every board file. But most did remove it explaining it was not working. It could be nobody run the rk339-nanpc-t4 dts from mainline so nobody disabled it. I you have such a board and can write to emmc with hs400 enabled that would help a lot. But I doubt it would work on nanopc-t4 while I saw tenths of boards. in 2023 so pretty late after the issue arised (maybe 2 years) Rock 4C+ https://github.com/torvalds/linux/commit/2bd1d2dd808c60532283e9cf05110bf1bf2f9079 Rock Pi 4 https://github.com/torvalds/linux/commit/cee572756aa2cb46e959e9797ad4b730b78a050b and all armbian rk3399 boards have it disabled as far as I know. by the way, thanks for the challenge to prove my point 🙂 . I found this https://github.com/torvalds/linux/commit/463be3cb357dab7d7e4d8dcc7c15c642e10c5bef arm64: dts: rockchip: add enable-strobe-pulldown to emmc phy on nanopi4 and this https://github.com/torvalds/linux/commit/8b5c2b45b8f0a11c9072da0f7baf9ee986d3151e phy: rockchip: set pulldown for strobe line in dts which might be related to this emmc issue (as noted in the mainline patch in the rockchip kernel this emmc pulldown setting is always enabled while in mainline it is disabled by default). This patch introduced in 5.11 might explain why even with the regulator issue fixed (the one I found by regression testing) with newer kernel I still had hs400 failing on me. Edit: indeed with this emmc_phy pulldown setting in the dts I can now write to the emmc in hs400es mode ! Thanks a lot for pointing out that nanopc-t4 was working with hs400es! The thing is they never posted a message binding the CQE error messages to their fix.. they tell it fixes a write error. And knowing how many hs400 broken leftover there was - including the two above commit from 2023 - this was pretty hard to guess they had a fix for their board dts. I believe the rationale is the same as for rk3588 "arm64: dts: rockchip: Fix eMMC Data Strobe PD on rk3588" https://github.com/torvalds/linux/commit/37f3d6108730713c411827ab4af764909f4dfc78 " JEDEC standard JESD84-B51 defines the eMMC Data Strobe line, which is currently used only in HS400 mode, as a device->host clock signal that "is used only in read operation. The Data Strobe is always High-Z (not driven by the device and pulled down by RDS) or Driven Low in write operation, except during CRC status response." RDS is a pull-down resistor specified in the 10K-100K ohm range. Thus per the standard, the Data Strobe is always pulled to ground (by the eMMC and/or RDS) during write operations. Evidently, the eMMC host controller in the RK3588 considers an active voltage on the eMMC-DS line during a write to be an error. The default (i.e. hardware reset, and Rockchip BSP) behavior for the RK3588 is to activate the eMMC-DS pin's builtin pull-down. As a result, many RK3588 board designers do not bother adding a dedicated RDS resistor, instead relying on the RK3588's internal bias. The current devicetree, however, disables this bias (`pcfg_pull_none`), breaking HS400-mode writes for boards without a dedicated RDS, but with an eMMC chip that chooses to High-Z (instead of drive-low) the eMMC-DS line. (The Turing RK1 is one such board.) Fix this by changing the bias in the (common) emmc_data_strobe case to reflect the expected hardware/BSP behavior. This is unlikely to cause regressions elsewhere: the pull-down is only relevant for High-Z eMMCs, and if this is redundant with a (dedicated) RDS resistor, the effective result is only a lower resistance to ground -- where the range of tolerance is quite high. If it does, it's better fixed in the specific devicetrees. except for rk3588 they decided to do the opposite, that is set the pulldown for all and let specific boards disabled it. So to fix hs400es on rk3399 add under the emmc_phy node on most (if not all) boards dts: rockchip,enable-strobe-pulldown;

sorry for the lag. I do not understand. You do not test the same device each time ? 50 seconds or one minute seems a lot, the last test with 1 second is fine. You should have had the same time for each test on the same enclosure and drive with "-d sat". Otherwise, something changed that is out of smartmontools scope. And I do not know what could affect "smartctl -i" time out of the "-d sat" (though there likely are other variables that could matter, but I don't know them). Did you mean by "more faster than the week ago" that nowadays it is always fast (ie 1 second)? @n3o do you have UAS is blacklisted for this device, using usb-storage instead in the dmesg logs for your usb device? else check usbstoragequircks as I told you in previous comments (/boot/armbianEnv.txt)

The helios64 patch to setup the sata power lines is already mainlined (line 92 ff). So it remains unclear why the sata LEDs are flashing at kernel startup. @ebin-dev yes and I thought that when the armbian dts for helios64 was migrated to be based on the mainline one this code was kept so I thought it was the cause of the new behavior. But inspecting the armbian patch more thoroughly removes the sata power lines mainline dts code. So the issue must be otherwise. I admit that I give a higher priority to the crashes I get because these flashing leds seem harmless.

switch to partitions #0, OK mmc1 is current device Scanning mmc 1:1... Found U-Boot script /boot/boot.scr 3185 bytes read in 6 ms (517.6 KiB/s) ## Executing script at 00500000 Boot script loaded from mmc 1 166 bytes read in 4 ms (40 KiB/s) 14541965 bytes read in 620 ms (22.4 MiB/s) Failed to load '/boot/Image' 86896 bytes read in 14 ms (5.9 MiB/s) 2698 bytes read in 10 ms (262.7 KiB/s) Applying kernel provided DT fixup script (rockchip-fixup.scr) ## Executing script at 09000000 Bad Linux ARM64 Image magic! SCRIPT FAILED: continuing... to me this looks like a file required by u-boot got corrupted during your "freeze". If mmc1 is an SD card you could mount it from a computer and check files like /boot/armbianEnv.txt and the kernel image and initrd are present in /boot. If kernel or initrd an issue (due to I suppose your freeze furing upgrade), best would be to chroot to the SD card and reinstall the linux-image package. An example of a valid /boot/armbianEnv.txt verbosity=7 bootlogo=false overlay_prefix=rockchip rootdev=UUID=a79a14c0-3cf4-4fb9-a6c6-838571351371 rootfstype=ext4 usbstoragequirks=0x2537:0x1066:u,0x2537:0x1068:u,0x0bc2:0x231a:u,0x1058:0x2621:u note the rootdev=UUID could vary,as the usbstoragequirks

Edit: I was wrong. as first the import of the upstream linux dts was done for 6.3 not 6.1 in armbian. Then checking the new armbian helios64 patchset it does remove this code to setup the sata power lines https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/arch/arm64/boot/dts/rockchip/rk3399-kobol-helios64.dts?id=8169b9894dbd2d4e440cfbc5fe9f733e5876a564 I woudl have to investigate why the sata lines are flashing at kernel startup. I do have these flashing leds. I have U-Boot 2022.07-armbian (Jul 21 2023 - 02:01:45 +0000).

All helios64 system that flashed armbian u-boot since it was switch to full mainline uboot has this issue. The only way not to have it is to not have flashed the fully mainlined u-boot (without rockchip DDR blob) or to never stress the ram. I have not sent the PR with the workaround (which is to build u-boot mainline with the rockhip DDR blob). Mind this is only a workaround. One should fix the upstream u-boot code that sets the LPDDR4 settings. One step would be to find out if other rk3399 boards with LPDDR4 (Nanopi M4 v2, Rock Pi 4, Orange Pi 4, etc). If one owns such a board it would be great to check if one can reproduce the issue with my test case and similar u-boot.

This is totally harmless. This is explained in the link you gave "If the kaslrseed command hasn't been compiled in to u-boot, it gracefully skips generating the kASLR".

@RmleonardI have similar crashes but have not found a simple reproducer. When you say you are able to reproduce with the board out, do you mean without anything plugged into it except the power adapter and ethernet? I was testing if it was not related to PCIe/SATA of the board but if you are able to reproduce without any disks plugged into the board that would help me phase out this option. Also, you don't remember if you updated the OS. But do you remember if you ever updated before? And if so when was the last update you remember doing?

@ebin-dev I do not understand what went wrong. Could you detail the issues ? On my side I also rstnc but the other way around, from emmc to SD, to have an exact copy of the emmc install. To be able to reproduce the issues I had (kernel crashes, that might be related to raid 10 high transfert rate on the rk3399 and pci-SATA bridge).

@ebin-dev well sorry I meant that emmc write with hs400 mode reenabked with a local patch is still broken. But indeed all rk3399 boards reverted to hs200 in kernel, Helios 64 first included the workaround (down to hs200) in Armbian 21.08.2. The first armbian helios64 kernel to ship the broken kernel was with the 5.10.60 Linux kernel. Emmc was and is stable in hs200 mode, my writing issue is with hs400 post 5.10.60.

@ebin-dev the armbian policy did not change, the status of the helios64 port did. It is now community maintained meaning the board packages are not provided by armbian (I believe if someone step up as maintainer, meaning he tests the armbian releases, helios64 armbian builds would be reumed).

Are you sure it was not hung after "Start kernel Give root password:" ? This is not uboot but the OS (probably feck failed and the OS booted in single use mode). Maybe the armbian fan script could be improved to lower the fan speed when entering single user mode. The only case when full fan speed could not be tweaked is when bootingbor shutting down with the default armbian fan config max speed at max and the kernel crashes. If the filesystems are not critical, to workaround the single user mode full fan speed waiting for you to fsck the filesystem, you could set fsck to attempt to repair all filesystem errors on its own with "fsck.repair=yes" https://www.systutorials.com/how-to-force-a-fsck-during-next-rebooting-of-linux/ I do so on a few boxes as either way, when I feck manually I reply yes to all questions ... but I have nothing that critical.

Ctrl+d is not supposed to boot without password. Just to boot with the default runlevel when you are in single user mode. Single user mode is the default runlevel when fsck failed or such bot errors. If you do not know the root password you can pass "emergency" to the kernel (or even more brute force "init=/bin/bash"). To pass this to the kernel you can halt the uboot boot, then enter: setenv extraboardargs "emergency" Then enter "boot". (Or "run bootcmd_mmc0/1 to force bootimg the Linux kernel from emmc or ad as I explained here Note the jumper is to force the helios64 to load uboot from either emmc or sd. From either of these uboot boots you can after you halter this uboot boot start the OS including the kernel from either emmc or sd via the above "run bootcmd_mmc0/1"). Booting u-boot and the OS from the same media is not hardcoded. Still you have to remove the jumper for the emmc to be visible from u-boot/OS. So you cannot do all possible combinations of jumper and uboot prompts. Note that you can also add the emergency or "init=/bin/bash" to the kernel commandd line by writinyto the OS partition /boot/armbianEnv.txt (for an SD card just moun tit on another computer and for the emmc you may be able to mount the emmc via usb-c via the P13 jumper iè recovery mode https://wiki.kobol.io/helios64/jumper/#usb-consolerecovery-mode-p13 but I never tried, it is told to enabled flashing, maybe one cannot mount)

It is perfectly fine but I believe you have to stop u-boot boot in between your jumper changes or at least that is what I do. To stop u-boot you press a key durimg uboot boot (I believe just before it detect the storages, better press the key multiple times during uboot boot). You press the key on the serial console. Then remove the jumper and enter "boot" on uboot prompt (I might be mistaken though I believe "boot" does the job, else search for "run bootcmd_mmc1" or "run bootcmd_mmc0" on the armbian forum, I believe in ubooc0 is the emmc and mmc1 is the sf card). This so the jumper is removed before the Linux kernel probe the hardware. You might be able to remove the jumper after boot and do the drive probes later with kpartx but I never tried. Also you can remove the jumper after uboot prompts and still boot to SD card then fsck and mount the emmc OS partition and then chroot to it and do whatever you want as if it was the current OS (note that before chroot you have to bind mount /dev, /sys and /prox from the current root partition to the emmc mount point, else any programs will not work).

This a user modifiable setting, though I trust the Kobold team choice to setup the armbian fan config to set the fan full speed at boot and shutdown. If you have nothing critical and nothing that gets hot in the helios64, you could lower the max speed in the config file (sorry I forgot it's location or if it can be tweaked by an arm Ian program)

Did you modify the governor and it's frequencies in armbian-config ? Else on reboot armbian boot scripts will reset to the armbian-config set value. In my case to avoid random successful boots because of timing I set the governor in /boot/armbianEnv.txt on the kernel command. This so I can better reproduce my bug and avoid success because the governor was set late on the boot sequence by the armbian scripts.

@BinaryWaves This looks like the uboot boot proceed so the kasrlseed command is like is told in https://github.com/armbian/build/pull/4352 , ignored if not supported by your uboot version. After "Starting kernel" you get "Give root password:". To me this means there was an error after the Linux kernel booted, probably the filesystem detected an error and you will have to login with the root password and fsck the filesystem manually. Reading the backlog it looks like you have the same state on the SD that you had with the emmc. As explained in the following post if you do not know the root password, pass "emergency" to the kernel command line. Though isn't the root password the one that you set at first boot (maybe it is with OMV installs only)? If not you should be able to find the default armbian root password on the forum.

If you have SD card you could install an image to the SD and chroot to the emmc install to diagnose (fsck, etc) I would have thought of a HDD filesystem repair but I do not think a Ctrl+D from a serial console could interrupt an fsck (but I never tried). Such an fsck can take a while. It could also be a mdadm raid resync or rebuild. You can check with cat /proc/mdstat When you said you borked your emmc install could you have pressed teh reset or poweroff button? That would point to an fsck/mdadm resync taking ages. What is your setup an mdadm raid? Which filesystem is your data on? I do not think the emmc OS filesystem is borked as Ctrl+D is enough to give you a working boot.

@n3o have you sorted out if the smartctl issue was due to the missing "-d sat" option? Do you have an ssh session to your helios64? As I told you could rebuild the bookworm smartmontools package for Bullseye if need be. It is a little bit involved though.

I believe your issue is that with the USB-C UGREEN 5 bay enclosure SMART data display in OMV times out after 4 minutes (and that you are afraid there could be corrupt files during transfer to the enclosure related to this issue). I don't know about such an issue (USB to SATA adapter be it RAID or not) taking more than 4 minutes to reply to a SMART data request. So I suggest you to login to the Helios64 (ssh) if you know how to do this and run smartctl on the UGREEN device smartctl -i </dev/...> where the device is /dev/sdc and /dev/sdd in the logs you posted (you could also use the more easily identifiable /dev/disk/by-id/<...device>. I see two unexpected things in the logs: usb 2-1.2: Product: Ugreen Storage Device (...) [ 242.742731] usb 2-1.2: reset SuperSpeed USB device number 4 using xhci-hcd [ 117.252293] GPT:Primary header thinks Alt. header is not at the end of the disk. [ 117.252310] GPT:27344699391 != 27344764927 [ 117.252318] GPT:Alternate GPT header not at the end of the disk. [ 117.252321] GPT:27344699391 != 27344764927 [ 117.252325] GPT: Use GNU Parted to correct GPT errors. [ 117.252344] sdc: sdc1 the issue seems to be related to smartmontools https://forum.odroid.com/viewtopic.php?t=37153 "reset SuperSpeed USB device number 3 using xhci-hcd" - only if smartmontools are installed . (note he at first misspelled smartmontools as sysmontools it seems). Note he tells his UGREEN USB3-SATA adapter works but it could be your enclosure as a different controller than yours. Was your usb-c cable to the enclosure provided with the enclosure? A power issue with a weak cable could also cause such a reset (though your external power adapter to the enclose likely removes the power issue so it leaves the cable) ? Likely an issue with your enclosure and smartmontools. Might be that there is a workaround in smartmontools that could be automatically set up by OpenMediaVault. Or also could be that smartmontools could implement a workaround for your enclosure. The kernel can implement workarounds for nonstandard controllers, which might help avoid smartmontools confusing the enclosure controller (ie disable UAS). You might as well contact UGREEN and/or the smartmontools mailing list. The bridge chip for the USB SATA bridge seems to be JMicron "Gen1 SATA 6Gb/s Bridge" (USB idVendor=152d, idProduct=0576). Which seems supported for a long time https://linux-hardware.org/index.php?id=usb:152d-0576 (also in BSD https://bsd-hardware.info/?id=usb:152d-0576) The chip is told to be JMS576 https://www.smartmontools.org/ticket/1428 This chip had a fix post smartmontools 7.2 https://www.smartmontools.org/ticket/1428 (probably 7.3) . You can try if the workaround works by calling the device with: smartctl -d sat -i /dev/sdc Note the "-d sat". About kernel workaround, you might want to try to force usb-storage driver instead of UAS driver in /boot/armbianEnv.txt usbstoragequirks=152d:0576:u from https://forums.raspberrypi.com/viewtopic.php?t=338883#p2029597 With this tweak in the kernel logs, you should now have "UAS is blacklisted for this device, using usb-storage instead" About the GPT header error, I do not expect these to explain a slowdown during SMART retrieval (as this is not related to the partitions). EDIT: your Armbian 22.02.1 is bullseye which has smartmontools 7.2. Once you have checked that the issue is no more with "smartctl -d sata -i <device>" you will know the culprit. You can either rebuild the smartmontools package with the patch above for JMS576 or plan an upgrade to bookworm (smartmontools 7.3 has the fix per https://www.smartmontools.org/browser/tags/RELEASE_7_3/smartmontools/drivedb.h#L5936 ) . You cannot directly install the bookworm package on bullseye because it depends on bookworm libc6. But you may be able to rebuild the bookworm source package on bullseye (did not check from https://sources.debian.org/src/smartmontools/7.3-1/debian/control/ if any of the build dependencies are only available on bookworm though).