prahal

You can easily tell if you run uboot with the DDR blob. Then the uboot serial output starts with: `DDR Version 1.25 20210517 In soft reset SRX channel 0 CS = 0` (The full mainline u-boot starts with a TPL message if I remind correctly). What might also matters is the ATF shipped with the u-boot deb. You can tweak which ATF release is used in the armbian build framework. I would also be interested in knowing the uboot and ATF version which is in linux-u-boot-edge-helios64_22.02.1_arm64.deb (I guess the DDR blob version is 1.25 as above as it does not seem to have been upgraded since a long time, but tell me if it is not 1.25). For uboot version you have a line about SPL on the serial output: `U-Boot SPL 2022.07-armbian (Dec 20 2023 - 09:16:29 +0000)` The ATF version is told after BL31: `NOTICE: BL31: lts-v2.8.8(release):armbian NOTICE: BL31: Built : 09:16:22, Dec 20 2023` Note the ATF runs at runtime, the kernels calls it. Also do you always run with serial attached? Just to check it is not related to my stability issues. I will not be able to test the r8152 stability as I have not even made the soldering fix for it (I have an early helios64 board). To be complete could you test the full mainline u-boot, ie latest before I reintroduced the DDR bin blob? To check if r8152 behaved the same before and after I added the rockchip DDR blob back?

Thank you for the feedback (I do not use the r8152). Do you mean you have issues with only the r8152 2,5Gb interface ? Or that this is the most obvious issue with the latest u-boot? Sorry I don't know but is linux-u-boot-edge-helios64_22.02.1_arm64.deb the fully mainline u-boot before I restored the rockchip DDR blob or a completely different u-boot (is it based on u-boot 2022.07 ?) I mean does the rockchip DDR blob restored cause w regression or do you mean even with that workaround the current u-boot is still less stable than a way older u-boot?

Thanks a lot for the testing work. I am uneasy about sending this push request upstream, maybe only in armbian at first. The proper fix could end up being inverting the pulldown default value for the initial commit https://github.com/torvalds/linux/commit/8b5c2b45b8f0a11c9072da0f7baf9ee986d3151e in mainline. As was done for rk3588 boards: "arm64: dts: rockchip: Fix eMMC Data Strobe PD on rk3588" https://github.com/torvalds/linux/commit/37f3d6108730713c411827ab4af764909f4dfc78 " With comment: JEDEC standard JESD84-B51 defines the eMMC Data Strobe line, which is currently used only in HS400 mode, as a device->host clock signal that "is used only in read operation. The Data Strobe is always High-Z (not driven by the device and pulled down by RDS) or Driven Low in write operation, except during CRC status response." RDS is a pull-down resistor specified in the 10K-100K ohm range. Thus per the standard, the Data Strobe is always pulled to ground (by the eMMC and/or RDS) during write operations. Evidently, the eMMC host controller in the RK3588 considers an active voltage on the eMMC-DS line during a write to be an error. The default (i.e. hardware reset, and Rockchip BSP) behavior for the RK3588 is to activate the eMMC-DS pin's builtin pull-down. As a result, many RK3588 board designers do not bother adding a dedicated RDS resistor, instead relying on the RK3588's internal bias. The current devicetree, however, disables this bias (`pcfg_pull_none`), breaking HS400-mode writes for boards without a dedicated RDS, but with an eMMC chip that chooses to High-Z (instead of drive-low) the eMMC-DS line. (The Turing RK1 is one such board.) Fix this by changing the bias in the (common) emmc_data_strobe case to reflect the expected hardware/BSP behavior. This is unlikely to cause regressions elsewhere: the pull-down is only relevant for High-Z eMMCs, and if this is redundant with a (dedicated) RDS resistor, the effective result is only a lower resistance to ground -- where the range of tolerance is quite high. If it does, it's better fixed in the specific devicetrees But that would requires board designers or the author of the above rk3588 commit to confirm the same is done on most rk3399 boards.

No need to remove this one (you can have both hs400 and hs200 enabled). Thank you for the feedback.

The mainline 5.11 patch "phy: rockchip: set pulldown for strobe line in dts" https://github.com/torvalds/linux/commit/8b5c2b45b8f0a11c9072da0f7baf9ee986d3151e (not backported to 5.10.y) seems to have broken most if not all rockchip board EMMC HS400 enhanced strobe support. This probably affects supported rockchip boards (at least rk3399 ones, but probably most). That is boards exhibit: [ 18.985162] mmc1: running CQE recovery [ 18.988056] ------------[ cut here ]------------ [ 18.988500] mmc1: cqhci: spurious TCN for tag 12 and the filesystem ends up corrupted on write attempt. Note that another bug about regulator core DEFERRED support (which might have produced the same issue hardware wise) was introduced in 5.10.43 (I had bisected it to https://github.com/torvalds/linux/commit/98e48cd9283dbac0e1445ee780889f10b3d1db6a "regulator: core: resolve supply for boot-on/always-on regulators"). But I was confident that it to be fixed in at least 6.1 by https://github.com/torvalds/linux/commit/8a866d527ac0441c0eb14a991fa11358b476b11d "regulator: core: Resolve supply name earlier to prevent double-init" (introduced in 6.1), still EMMC was still failing on me. Thanks to @RussianNeuroMancer telling me that not all rk339 boards had EMMC HS400es broken, I found that nanopc-t4 had https://github.com/torvalds/linux/commit/463be3cb357dab7d7e4d8dcc7c15c642e10c5bef arm64: dts: rockchip: add enable-strobe-pulldown to emmc phy on nanopi4 So the current way, from this nanopc-t4 commit, to fix EMMC HS400 on most rockchip is to add "rockchip,enable-strobe-pulldown;" to the "emmc_phy" node (at least this node alias for rk3399). &emmc_phy { + rockchip,enable-strobe-pulldown; status = "okay"; }; With this patch I can renable hs400es for rk3399 emmc on helios64 (it is already set for nanopc-t4 in mainline). Details in: I believe the https://github.com/torvalds/linux/commit/8b5c2b45b8f0a11c9072da0f7baf9ee986d3151e mainline commit was wrong in that it probably should have done the opposite (that is enable the pulldown) as most boards were hardwired so. As was done for rk3588 boards: "arm64: dts: rockchip: Fix eMMC Data Strobe PD on rk3588" https://github.com/torvalds/linux/commit/37f3d6108730713c411827ab4af764909f4dfc78 " JEDEC standard JESD84-B51 defines the eMMC Data Strobe line, which is currently used only in HS400 mode, as a device->host clock signal that "is used only in read operation. The Data Strobe is always High-Z (not driven by the device and pulled down by RDS) or Driven Low in write operation, except during CRC status response." RDS is a pull-down resistor specified in the 10K-100K ohm range. Thus per the standard, the Data Strobe is always pulled to ground (by the eMMC and/or RDS) during write operations. Evidently, the eMMC host controller in the RK3588 considers an active voltage on the eMMC-DS line during a write to be an error. The default (i.e. hardware reset, and Rockchip BSP) behavior for the RK3588 is to activate the eMMC-DS pin's builtin pull-down. As a result, many RK3588 board designers do not bother adding a dedicated RDS resistor, instead relying on the RK3588's internal bias. The current devicetree, however, disables this bias (`pcfg_pull_none`), breaking HS400-mode writes for boards without a dedicated RDS, but with an eMMC chip that chooses to High-Z (instead of drive-low) the eMMC-DS line. (The Turing RK1 is one such board.) Fix this by changing the bias in the (common) emmc_data_strobe case to reflect the expected hardware/BSP behavior. This is unlikely to cause regressions elsewhere: the pull-down is only relevant for High-Z eMMCs, and if this is redundant with a (dedicated) RDS resistor, the effective result is only a lower resistance to ground -- where the range of tolerance is quite high. If it does, it's better fixed in the specific devicetrees. Maybe one can confirm this is the case not only for rk5588 but for other rockchip boards? (about the default for hardware reset and rockchip BSP with regards to active that eMMC-DS pin's builtin pulldown if any, and board designers for other boards than rk5588 also not bothering to add a dedicated RDS resistor, instead relying on the rockchip internal bias, also if any on non rk5588) At least two other boards disabled hs400es in mainline probbaly due to this patch disabling the internal pulldown by default "phy: rockchip: set pulldown for strobe line in dts" https://github.com/torvalds/linux/commit/8b5c2b45b8f0a11c9072da0f7baf9ee986d3151e in 2023 in vanilla Linux: Rock 4C+ https://github.com/torvalds/linux/commit/2bd1d2dd808c60532283e9cf05110bf1bf2f9079 Rock Pi 4 https://github.com/torvalds/linux/commit/cee572756aa2cb46e959e9797ad4b730b78a050b

@BinaryWaves see my post above for how to restore the files u-boot needs to start the kernel I have not yet been able to find out the cause of the random (sometimes often, sometimes months apart, crashes of the board). However, such crashes tend to corrupt files more easily on SD and EMMC than on HDD probably due to the block size. So you can end up unable to boot after such crashes due to file corruption. I hope one day to get rid of at least one such crash cause (if more than one cause is at stake) but not there yet. So one has to resort to chroot to the SD or EMMC and reinstall. Or reinstall a new image if one does not want to bother learning the required steps for chroot. You could also take OS FS backup images regularly until the issue is sorted out and reimage after a crash that broke boot (though they are pretty rare, so one might end up stopping imaging until it happens).

I can no longer reproduce this rescan behavior (LEDs stay solid blue before triggering on disk accesses). It could have been fixed days ago. I am currently on 6.6.7-edge-rockchip64 fetch and build on Armbian three days ago. U-Boot 2022.07-armbian (Jul 21 2023 - 02:01:45 +0000)

6.6 might be unstable, you could try the armbian repo 6.2 , that is linux-image-edge-rockchip64, linux-headers-edge-rockchip64 and linux-dtb-edge-rockchip64 at version 23.11.1 (mind if you have both current and edge kernel there are issues when a current upgrade is push while you have a edge version previously installed, this is an armbian packaging issue though. BUt for the matter of testing it is fine). Then ssh to the helios64 and edit the current dts with "armbian-config > System > Dtc" emmc_phy settings are in the "phy@f780" node (the default editor is nano, you can search with Ctrl+W). Add " rockchip,enable-strobe-pulldown; " under this node then under the mmc@fe330000 node add: mmc-hs400-1_8v; mmc-hs400-enhanced-strobe; confirm and reboot. It could be that the pulldown setting is enough even before linux 6.1 (there were other core regulator fixes, but the most important one was added in 6.1), so you could try with lower kernel version. root@helios64:~# hdparm -tT /dev/mmcblk0 /dev/mmcblk0: Timing cached reads: 1888 MB in 2.00 seconds = 944.72 MB/sec Timing buffered disk reads: 552 MB in 3.00 seconds = 183.86 MB/sec Note that EMMC on 6.6.7 /dev/mmcblk0 and SD /dev/mmcblk1 hdparm -tT /dev/mmcblk1 /dev/mmcblk1: Timing cached reads: 1858 MB in 2.00 seconds = 930.17 MB/sec Timing buffered disk reads: 128 MB in 3.10 seconds = 41.34 MB/sec

@piter75@ebin-dev Feel free to confirm the rockchip emmc phy pulldown dts indeed fixes emmc write in hs400es mode (I believe on most rk3399 boards, not only helios64). Likely on kernel up and above 6.1 (for the regulator fixes). Thanks to @RussianNeuroMancer I found: arm64: dts: rockchip: add enable-strobe-pulldown to emmc phy on nanopi4 https://github.com/torvalds/linux/commit/463be3cb357dab7d7e4d8dcc7c15c642e10c5bef and: phy: rockchip: set pulldown for strobe line in dts https://github.com/torvalds/linux/commit/8b5c2b45b8f0a11c9072da0f7baf9ee986d3151e which might be related to this emmc issue (as noted in the mainline patch in the rockchip kernel this emmc pulldown setting is always enabled while in mainline it is disabled by default). This patch was introduced in 5.11 which might explain why even with the regulator issue fixed (the one I found by regression testing) with kernel above 5.11 I still had hs400 failing on me. With this emmc_phy pulldown setting in the dts I can now write to the emmc in hs400es mode ! means one have to add "rockchip,enable-strobe-pulldown;" in the emmc_phy node for most boards --- arch/arm64/boot/dts/rockchip/rk3399-kobol-helios64.dts | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/arch/arm64/boot/dts/rockchip/rk3399-kobol-helios64.dts b/arch/arm64/boot/dts/rockchip/rk3399-kobol-helios64.dts index 69d76dea35d0..3c1965660fbd 100644 --- a/arch/arm64/boot/dts/rockchip/rk3399-kobol-helios64.dts +++ b/arch/arm64/boot/dts/rockchip/rk3399-kobol-helios64.dts @@ -436,10 +436,11 @@ &cpu_b0 { &cpu_b1 { cpu-supply = <&vdd_cpu_b>; }; &emmc_phy { + rockchip,enable-strobe-pulldown; status = "okay"; }; &gmac { assigned-clocks = <&cru SCLK_RMII_SRC>; @@ -965,13 +966,12 @@ &saradc { &sdhci { assigned-clock-rates = <150000000>; bus-width = <8>; mmc-hs200-1_8v; - // hs400 is broken on Helios64 since 5.10.60 - // mmc-hs400-1_8v; - // mmc-hs400-enhanced-strobe; + mmc-hs400-1_8v; + mmc-hs400-enhanced-strobe; supports-emmc; non-removable; disable-wp; status = "okay"; vqmmc-supply = <&vcc1v8_sys_s0>; -- There is also the regulator fix. Back then I bisected the first bad commit 06653ebc0ad2e0b7d799cd71a5c2933ed2fb7a66 which seems to have been a backport of the master branch to commit https://github.com/torvalds/linux/commit/98e48cd9283dbac0e1445ee780889f10b3d1db6a "regulator: core: resolve supply for boot-on/always-on regulators" introduced in 5.13. I am pretty confident the regulator issues were fixed at least in linux 6.1. Note that the emmc phy pulldown code was not backported to 5.10.y (https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/log/drivers/phy/rockchip/phy-rockchip-emmc.c?h=linux-5.10.y) So I believe it was first included in 5.11. I no longer have the following CQE errors when writing to emmc in hs400es mode: [ 18.985162] mmc1: running CQE recovery [ 18.988056] ------------[ cut here ]------------ [ 18.988500] mmc1: cqhci: spurious TCN for tag 12 [ 18.989019] WARNING: CPU: 0 PID: 269 at drivers/mmc/host/cqhci-core.c:787 cqhci_irq+0x4b4/0x640 [ 18.989838] Modules linked in: quota_v2 quota_tree r8152 snd_soc_hdmi_codec ftdi_sio usbserial snd_soc_rockchip_i2s leds_pwm snd_soc_rockchip_pcm snd_soc_core snd_pcm_dmaengine snd_pcm pwm_fan gpio_charger snd_timer panfrost gpu_sched snd hantro_vpu(C) soundcore rockchip_vdec(C) v4l2_h264 rockchip_rga rockchip_iep videobuf2_dma_contig videobuf2_vmalloc videobuf2_dma_sg v4l2_mem2mem sg videobuf2_memops videobuf2_v4l2 videobuf2_common fusb302 videodev tcpm typec mc gpio_beeper cpufreq_dt nfsd dm_mod auth_rpcgss nfs_acl lockd grace ledtrig_netdev sunrpc lm75 ip_tables x_tables autofs4 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx raid1 raid0 multipath linear raid10 uas realtek md_mod dwmac_rk stmmac_platform stmmac pcs_xpcs adc_keys [ 18.995876] CPU: 0 PID: 269 Comm: kworker/0:1H Tainted: G C 5.15.29-rockchip64 #trunk.0010 [ 18.996740] Hardware name: Helios64 (DT) [ 18.997104] Workqueue: kblockd blk_mq_run_work_fn [ 18.997555] pstate: 800000c5 (Nzcv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 18.998182] pc : cqhci_irq+0x4b4/0x640 [ 18.998537] lr : cqhci_irq+0x4b4/0x640 [ 18.998880] sp : ffff800008003d10 [ 18.999179] x29: ffff800008003d10 x28: ffff0000012b8e80 x27: ffff000004c2d580 [ 18.999819] x26: ffff00000652fc98 x25: ffff8000094d0888 x24: ffff800009c975e8 [ 19.000462] x23: ffff8000094f49c8 x22: 0000000000000002 x21: ffff000004c2d000 [ 19.001107] x20: 000000000000000c x19: ffff00000652fc80 x18: 0000000000000010 [ 19.001754] x17: ffff8000ee03c000 x16: ffff800008004000 x15: 00000000000002f6 [ 19.002400] x14: ffff800008003a20 x13: 00000000ffffffea x12: ffff800009abfbc8 [ 19.003047] x11: 0000000000000003 x10: ffff800009aa7b88 x9 : ffff800009aa7be0 [ 19.003694] x8 : 0000000000017fe8 x7 : c0000000ffffefff x6 : 0000000000000001 [ 19.004338] x5 : ffff8000ee03c000 x4 : 0000000000000000 x3 : 0000000000010004 [ 19.004980] x2 : 0000000000010003 x1 : f94ce7e57b395a00 x0 : 0000000000000000 [ 19.005623] Call trace: [ 19.005852] cqhci_irq+0x4b4/0x640 [ 19.006169] sdhci_arasan_cqhci_irq+0x5c/0x88 [ 19.006564] sdhci_irq+0xcc/0x10c0 [ 19.006876] __handle_irq_event_percpu+0x60/0x250 [ 19.007302] handle_irq_event_percpu+0x38/0x90 [ 19.007705] handle_irq_event+0x48/0xe8 [ 19.008058] handle_fasteoi_irq+0xb8/0x148 [ 19.008430] handle_domain_irq+0x90/0xd8 [ 19.008788] gic_handle_irq+0xb8/0x138 [ 19.009131] call_on_irq_stack+0x28/0x50 [ 19.009492] do_interrupt_handler+0x58/0x68 [ 19.009874] el1_interrupt+0x30/0x78 [ 19.010202] el1h_64_irq_handler+0x18/0x28 [ 19.010575] el1h_64_irq+0x74/0x78 [ 19.010889] preempt_count_sub+0x34/0xc0 [ 19.011251] _raw_spin_unlock_irqrestore+0x20/0x40 [ 19.011686] sdhci_cqe_enable+0x130/0x228 [ 19.012060] sdhci_arasan_cqe_enable+0x94/0xb8 [ 19.012465] cqhci_request+0xd0/0x650 [ 19.012808] mmc_cqe_start_req+0xb4/0x198 [ 19.013177] mmc_blk_mq_issue_rq+0x494/0x9a8 [ 19.013567] mmc_mq_queue_rq+0x114/0x2b0 [ 19.013928] blk_mq_dispatch_rq_list+0x120/0x7e8 [ 19.014355] __blk_mq_sched_dispatch_requests+0xc4/0x1e0 [ 19.014835] blk_mq_sched_dispatch_requests+0x3c/0x78 [ 19.015291] __blk_mq_run_hw_queue+0x64/0xa0 [ 19.015678] blk_mq_run_work_fn+0x20/0x30 [ 19.016042] process_one_work+0x20c/0x4c8 [ 19.016407] worker_thread+0x48/0x478 [ 19.016739] kthread+0x138/0x150 [ 19.017033] ret_from_fork+0x10/0x20 [ 19.017358] ---[ end trace 70e1eee6af816777 ]--- [ 19.020419] mmc1: running CQE recovery [ 19.025922] mmc1: running CQE recovery [ 19.028135] blk_update_request: I/O error, dev mmcblk1, sector 1605640 op 0x1:(WRITE) flags 0x800 phys_seg 57 prio class 0 [ 19.029190] Buffer I/O error on dev mmcblk1p1, logical block 196609, lost sync page write [ 19.029989] EXT4-fs error (device mmcblk1p1): ext4_check_bdev_write_error:218: comm sed: Error while async write back metadata [ 19.030191] Aborting journal on device mmcblk1p1-8. [ 19.032803] EXT4-fs (mmcblk1p1): Remounting filesystem read-only [ 19.033426] EXT4-fs error (device mmcblk1p1) in __ext4_new_inode:1136: Journal has aborted Feel free to submit this patch to Armbian github or even to the mainline helios64 dts on your side.

@ebin-dev note that u-boot is not updated in the device when the u-boot package is updated. I really should have pushed the workaround since I cooked it up, but apt upgrade cannot break the u-boot on the board. I will try to send the pull request for this u-boot workaround in January. Note also that it is only a workaround. The support for LPDDR4 on u-boot mainline might be buggy. But I don't know how I can fix it on my side. The rockchip blob that works is a binary without sources. Maybe one could ask rockchip devs to cook a fix for u-boot mainline?

What do you mean? hs400 is still there. I did not check every board file. But most did remove it explaining it was not working. It could be nobody run the rk339-nanpc-t4 dts from mainline so nobody disabled it. I you have such a board and can write to emmc with hs400 enabled that would help a lot. But I doubt it would work on nanopc-t4 while I saw tenths of boards. in 2023 so pretty late after the issue arised (maybe 2 years) Rock 4C+ https://github.com/torvalds/linux/commit/2bd1d2dd808c60532283e9cf05110bf1bf2f9079 Rock Pi 4 https://github.com/torvalds/linux/commit/cee572756aa2cb46e959e9797ad4b730b78a050b and all armbian rk3399 boards have it disabled as far as I know. by the way, thanks for the challenge to prove my point 🙂 . I found this https://github.com/torvalds/linux/commit/463be3cb357dab7d7e4d8dcc7c15c642e10c5bef arm64: dts: rockchip: add enable-strobe-pulldown to emmc phy on nanopi4 and this https://github.com/torvalds/linux/commit/8b5c2b45b8f0a11c9072da0f7baf9ee986d3151e phy: rockchip: set pulldown for strobe line in dts which might be related to this emmc issue (as noted in the mainline patch in the rockchip kernel this emmc pulldown setting is always enabled while in mainline it is disabled by default). This patch introduced in 5.11 might explain why even with the regulator issue fixed (the one I found by regression testing) with newer kernel I still had hs400 failing on me. Edit: indeed with this emmc_phy pulldown setting in the dts I can now write to the emmc in hs400es mode ! Thanks a lot for pointing out that nanopc-t4 was working with hs400es! The thing is they never posted a message binding the CQE error messages to their fix.. they tell it fixes a write error. And knowing how many hs400 broken leftover there was - including the two above commit from 2023 - this was pretty hard to guess they had a fix for their board dts. I believe the rationale is the same as for rk3588 "arm64: dts: rockchip: Fix eMMC Data Strobe PD on rk3588" https://github.com/torvalds/linux/commit/37f3d6108730713c411827ab4af764909f4dfc78 " JEDEC standard JESD84-B51 defines the eMMC Data Strobe line, which is currently used only in HS400 mode, as a device->host clock signal that "is used only in read operation. The Data Strobe is always High-Z (not driven by the device and pulled down by RDS) or Driven Low in write operation, except during CRC status response." RDS is a pull-down resistor specified in the 10K-100K ohm range. Thus per the standard, the Data Strobe is always pulled to ground (by the eMMC and/or RDS) during write operations. Evidently, the eMMC host controller in the RK3588 considers an active voltage on the eMMC-DS line during a write to be an error. The default (i.e. hardware reset, and Rockchip BSP) behavior for the RK3588 is to activate the eMMC-DS pin's builtin pull-down. As a result, many RK3588 board designers do not bother adding a dedicated RDS resistor, instead relying on the RK3588's internal bias. The current devicetree, however, disables this bias (`pcfg_pull_none`), breaking HS400-mode writes for boards without a dedicated RDS, but with an eMMC chip that chooses to High-Z (instead of drive-low) the eMMC-DS line. (The Turing RK1 is one such board.) Fix this by changing the bias in the (common) emmc_data_strobe case to reflect the expected hardware/BSP behavior. This is unlikely to cause regressions elsewhere: the pull-down is only relevant for High-Z eMMCs, and if this is redundant with a (dedicated) RDS resistor, the effective result is only a lower resistance to ground -- where the range of tolerance is quite high. If it does, it's better fixed in the specific devicetrees. except for rk3588 they decided to do the opposite, that is set the pulldown for all and let specific boards disabled it. So to fix hs400es on rk3399 add under the emmc_phy node on most (if not all) boards dts: rockchip,enable-strobe-pulldown;

sorry for the lag. I do not understand. You do not test the same device each time ? 50 seconds or one minute seems a lot, the last test with 1 second is fine. You should have had the same time for each test on the same enclosure and drive with "-d sat". Otherwise, something changed that is out of smartmontools scope. And I do not know what could affect "smartctl -i" time out of the "-d sat" (though there likely are other variables that could matter, but I don't know them). Did you mean by "more faster than the week ago" that nowadays it is always fast (ie 1 second)? @n3o do you have UAS is blacklisted for this device, using usb-storage instead in the dmesg logs for your usb device? else check usbstoragequircks as I told you in previous comments (/boot/armbianEnv.txt)

The helios64 patch to setup the sata power lines is already mainlined (line 92 ff). So it remains unclear why the sata LEDs are flashing at kernel startup. @ebin-dev yes and I thought that when the armbian dts for helios64 was migrated to be based on the mainline one this code was kept so I thought it was the cause of the new behavior. But inspecting the armbian patch more thoroughly removes the sata power lines mainline dts code. So the issue must be otherwise. I admit that I give a higher priority to the crashes I get because these flashing leds seem harmless.

switch to partitions #0, OK mmc1 is current device Scanning mmc 1:1... Found U-Boot script /boot/boot.scr 3185 bytes read in 6 ms (517.6 KiB/s) ## Executing script at 00500000 Boot script loaded from mmc 1 166 bytes read in 4 ms (40 KiB/s) 14541965 bytes read in 620 ms (22.4 MiB/s) Failed to load '/boot/Image' 86896 bytes read in 14 ms (5.9 MiB/s) 2698 bytes read in 10 ms (262.7 KiB/s) Applying kernel provided DT fixup script (rockchip-fixup.scr) ## Executing script at 09000000 Bad Linux ARM64 Image magic! SCRIPT FAILED: continuing... to me this looks like a file required by u-boot got corrupted during your "freeze". If mmc1 is an SD card you could mount it from a computer and check files like /boot/armbianEnv.txt and the kernel image and initrd are present in /boot. If kernel or initrd an issue (due to I suppose your freeze furing upgrade), best would be to chroot to the SD card and reinstall the linux-image package. An example of a valid /boot/armbianEnv.txt verbosity=7 bootlogo=false overlay_prefix=rockchip rootdev=UUID=a79a14c0-3cf4-4fb9-a6c6-838571351371 rootfstype=ext4 usbstoragequirks=0x2537:0x1066:u,0x2537:0x1068:u,0x0bc2:0x231a:u,0x1058:0x2621:u note the rootdev=UUID could vary,as the usbstoragequirks

Edit: I was wrong. as first the import of the upstream linux dts was done for 6.3 not 6.1 in armbian. Then checking the new armbian helios64 patchset it does remove this code to setup the sata power lines https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/arch/arm64/boot/dts/rockchip/rk3399-kobol-helios64.dts?id=8169b9894dbd2d4e440cfbc5fe9f733e5876a564 I woudl have to investigate why the sata lines are flashing at kernel startup. I do have these flashing leds. I have U-Boot 2022.07-armbian (Jul 21 2023 - 02:01:45 +0000).

All helios64 system that flashed armbian u-boot since it was switch to full mainline uboot has this issue. The only way not to have it is to not have flashed the fully mainlined u-boot (without rockchip DDR blob) or to never stress the ram. I have not sent the PR with the workaround (which is to build u-boot mainline with the rockhip DDR blob). Mind this is only a workaround. One should fix the upstream u-boot code that sets the LPDDR4 settings. One step would be to find out if other rk3399 boards with LPDDR4 (Nanopi M4 v2, Rock Pi 4, Orange Pi 4, etc). If one owns such a board it would be great to check if one can reproduce the issue with my test case and similar u-boot.

This is totally harmless. This is explained in the link you gave "If the kaslrseed command hasn't been compiled in to u-boot, it gracefully skips generating the kASLR".

@RmleonardI have similar crashes but have not found a simple reproducer. When you say you are able to reproduce with the board out, do you mean without anything plugged into it except the power adapter and ethernet? I was testing if it was not related to PCIe/SATA of the board but if you are able to reproduce without any disks plugged into the board that would help me phase out this option. Also, you don't remember if you updated the OS. But do you remember if you ever updated before? And if so when was the last update you remember doing?

@ebin-dev I do not understand what went wrong. Could you detail the issues ? On my side I also rstnc but the other way around, from emmc to SD, to have an exact copy of the emmc install. To be able to reproduce the issues I had (kernel crashes, that might be related to raid 10 high transfert rate on the rk3399 and pci-SATA bridge).

@ebin-dev well sorry I meant that emmc write with hs400 mode reenabked with a local patch is still broken. But indeed all rk3399 boards reverted to hs200 in kernel, Helios 64 first included the workaround (down to hs200) in Armbian 21.08.2. The first armbian helios64 kernel to ship the broken kernel was with the 5.10.60 Linux kernel. Emmc was and is stable in hs200 mode, my writing issue is with hs400 post 5.10.60.

@ebin-dev the armbian policy did not change, the status of the helios64 port did. It is now community maintained meaning the board packages are not provided by armbian (I believe if someone step up as maintainer, meaning he tests the armbian releases, helios64 armbian builds would be reumed).

Are you sure it was not hung after "Start kernel Give root password:" ? This is not uboot but the OS (probably feck failed and the OS booted in single use mode). Maybe the armbian fan script could be improved to lower the fan speed when entering single user mode. The only case when full fan speed could not be tweaked is when bootingbor shutting down with the default armbian fan config max speed at max and the kernel crashes. If the filesystems are not critical, to workaround the single user mode full fan speed waiting for you to fsck the filesystem, you could set fsck to attempt to repair all filesystem errors on its own with "fsck.repair=yes" https://www.systutorials.com/how-to-force-a-fsck-during-next-rebooting-of-linux/ I do so on a few boxes as either way, when I feck manually I reply yes to all questions ... but I have nothing that critical.

Ctrl+d is not supposed to boot without password. Just to boot with the default runlevel when you are in single user mode. Single user mode is the default runlevel when fsck failed or such bot errors. If you do not know the root password you can pass "emergency" to the kernel (or even more brute force "init=/bin/bash"). To pass this to the kernel you can halt the uboot boot, then enter: setenv extraboardargs "emergency" Then enter "boot". (Or "run bootcmd_mmc0/1 to force bootimg the Linux kernel from emmc or ad as I explained here Note the jumper is to force the helios64 to load uboot from either emmc or sd. From either of these uboot boots you can after you halter this uboot boot start the OS including the kernel from either emmc or sd via the above "run bootcmd_mmc0/1"). Booting u-boot and the OS from the same media is not hardcoded. Still you have to remove the jumper for the emmc to be visible from u-boot/OS. So you cannot do all possible combinations of jumper and uboot prompts. Note that you can also add the emergency or "init=/bin/bash" to the kernel commandd line by writinyto the OS partition /boot/armbianEnv.txt (for an SD card just moun tit on another computer and for the emmc you may be able to mount the emmc via usb-c via the P13 jumper iè recovery mode https://wiki.kobol.io/helios64/jumper/#usb-consolerecovery-mode-p13 but I never tried, it is told to enabled flashing, maybe one cannot mount)

It is perfectly fine but I believe you have to stop u-boot boot in between your jumper changes or at least that is what I do. To stop u-boot you press a key durimg uboot boot (I believe just before it detect the storages, better press the key multiple times during uboot boot). You press the key on the serial console. Then remove the jumper and enter "boot" on uboot prompt (I might be mistaken though I believe "boot" does the job, else search for "run bootcmd_mmc1" or "run bootcmd_mmc0" on the armbian forum, I believe in ubooc0 is the emmc and mmc1 is the sf card). This so the jumper is removed before the Linux kernel probe the hardware. You might be able to remove the jumper after boot and do the drive probes later with kpartx but I never tried. Also you can remove the jumper after uboot prompts and still boot to SD card then fsck and mount the emmc OS partition and then chroot to it and do whatever you want as if it was the current OS (note that before chroot you have to bind mount /dev, /sys and /prox from the current root partition to the emmc mount point, else any programs will not work).

This a user modifiable setting, though I trust the Kobold team choice to setup the armbian fan config to set the fan full speed at boot and shutdown. If you have nothing critical and nothing that gets hot in the helios64, you could lower the max speed in the config file (sorry I forgot it's location or if it can be tweaked by an arm Ian program)