KingJ Posted Sunday at 05:37 PM Posted Sunday at 05:37 PM I've recently switched my FriendlyElec NanoPC-T6 to use the Armbian Linux v6.12 server image, built on the 24th of May 2025. Booting of SD works fine, but when installing to the eMMC chip some I/O errors can be found in the kernel logs: [ 151.814773] I/O error, dev mmcblk0, sector 176 op 0x0:(READ) flags 0x80700 phys_seg 10 prio class 2 This happens when under heavy I/O load - e.g. performing an apt upgrade. I ran badblocks over the entire eMMC chip without issue - but that puts a much lower strain on the eMMC. Therefore, i'm convinced that the eMMC chip itself is fine. Poking around a bit, this seems to be because the A3A444 eMMC chip which some NanoPC-T6 SBCs shipped with do not support HS400 mode properly. There is an OpenWRT bug report about this, which fixed the issue by patching the dtsi to force HS200 mode. My kernel logs confirm that I have the A3A444 eMMC chip, and that it is currently running in HS400 mode: sudo dmesg -e | grep -i mmc [ +0.015143] mmc0: SDHCI controller on fe2e0000.mmc [fe2e0000.mmc] using ADMA [ +0.051640] dwmmc_rockchip fe2c0000.mmc: IDMAC supports 32-bit address mode. [ +0.000020] dwmmc_rockchip fe2c0000.mmc: Using internal DMA controller. [ +0.000008] dwmmc_rockchip fe2c0000.mmc: Version ID is 270a [ +0.000025] dwmmc_rockchip fe2c0000.mmc: DW MMC controller at irq 91,32 bit host data width,256 deep fifo [ +0.000210] dwmmc_rockchip fe2c0000.mmc: Got CD GPIO [ +0.012806] mmc_host mmc1: Bus speed (slot 0) = 400000Hz (slot req 400000Hz, actual 400000HZ div = 0) [ +0.007795] mmc0: new HS400 Enhanced strobe MMC card at address 0001 [ +0.000906] mmcblk0: mmc0:0001 A3A444 230 GiB [ +0.004124] mmcblk0: p1 [ +0.000720] mmcblk0boot0: mmc0:0001 A3A444 4.00 MiB [ +0.001831] mmcblk0boot1: mmc0:0001 A3A444 4.00 MiB [ +0.001742] mmcblk0rpmb: mmc0:0001 A3A444 4.00 MiB, chardev (243:0) [ +0.227497] EXT4-fs (mmcblk0p1): mounted filesystem a4f48be8-f667-4cac-a9a7-61ce8f9035d1 ro with ordered data mode. Quota mode: none. [ +0.021022] EXT4-fs (mmcblk0p1): re-mounted a4f48be8-f667-4cac-a9a7-61ce8f9035d1 r/w. I've tried to use a user device overlay to use mmc-hs200-1_8v, but this doesn't appear to work. I think this is because Device Overlays can only replace elements or add to the tree? i.e. I cannot use an overlay to remove the existing mmc-hs400-1_8v; and mmc-hs400-enhanced-strobe; entries. Is there a way to do this with a user device overlay, or will I need to try to add a similar patch to OpenWRT's one in the kernel? This is the first time i've used Armbian so i'm a little unsure about how i'd go about doing the latter. I've found a similar report of this issue from a couple of years ago by @SuperKali, albeit with no resolution. However, I can see they're now listed as one of the community maintainers for this board so i'm hoping it's OK to mention them in this thread to see if they know how best to force HS200 mode for the eMMC! 0 Quote
KingJ Posted Monday at 08:35 PM Author Posted Monday at 08:35 PM I think i've managed to work around this, and set HS200 mode. I did this by decompiling /boot/dtb/rockchip/rk3588-nanopc-t6.dtb, modifying the mmc@fe2e0000 section to remove the mmc-hs400-1_8v; and mmc-hs400-enhanced-strobe; lines, adding a new mmc-hs200-1_8v; line and then recompiling it again. dtc -I dtb -O dts -f /boot/dtb/rockchip/rk3588-nanopc-t6.dtb -o rk3588-nanopc-t6.dts # Modify rk3588-nanopc-t6.dts here dtc -I dts -O dtb rk3588-nanopc-t6.dts -o /boot/dtb/rockchip/rk3588-nanopc-t6.dtb After rebooting, I confirmed that it was now running in HS200 mode: [ +0.010753] mmc0: SDHCI controller on fe2e0000.mmc [fe2e0000.mmc] using ADMA [ +0.034734] mmc0: new HS200 MMC card at address 0001 [ +0.000571] mmcblk0: mmc0:0001 A3A444 230 GiB I ran a heavy random read/write test, and observed no errors. However, I think this will be lost during a future update - so i'll see if I can figure out how to submit a patch! 0 Quote
KingJ Posted Monday at 09:37 PM Author Posted Monday at 09:37 PM Raised up a PR with a patch. 🤞 0 Quote
usual user Posted Monday at 11:17 PM Posted Monday at 11:17 PM 1 hour ago, KingJ said: Raised up a PR with a patch. 🤞 You will certainly gain many grateful users whose devices are not equipped with an affected eMMC, but who still have to suffer from the slowdown nonetheless. Does the attached overlay work for you?rk3588-nanopc-t6-emmc.dtbo 0 Quote
KingJ Posted Monday at 11:56 PM Author Posted Monday at 11:56 PM Right, that's the downside - anyone with a different eMMC chip will see a drop in performance. I've just given that overlay a try - thank you for suggesting it. I put it in to /boot/overlay-user, then edited /boot/armbianEnv.txt to include the line user_overlays=rk3588-nanopc-t6-emmc . Unfortunately, it then failed to boot. I had to put in an SD card, boot from that, mount the eMMC partition, and then remove the overload from armbianEnv.txt to get it to boot successfully from the eMMC again. 0 Quote
usual user Posted yesterday at 06:27 PM Posted yesterday at 06:27 PM 18 hours ago, KingJ said: Unfortunately, it then failed to boot. Admittedly, I had not tested the DTBO at runtime so far, but only applied it statically to the base DTB and checked whether all desired changes were made as intended. Now I am running my device with the applied overlay and I get the following: [ 0.940862] mmc0: new HS200 MMC card at address 0001 [ 0.941665] mmcblk0: mmc0:0001 A3A561 57.6 GiB [ 0.943039] mmcblk0: p1 p2 [ 0.943767] mmcblk0boot0: mmc0:0001 A3A561 4.00 MiB [ 0.945199] mmcblk0boot1: mmc0:0001 A3A561 4.00 MiB [ 0.946661] mmcblk0rpmb: mmc0:0001 A3A561 16.0 MiB, chardev (506:0) So everything is as expected, and yes, I have ensured that my system is running with the applied overlay. In the past, I have at least once noticed far too late that my system was running in a fallback, and a feature to be tested was not applied at all. That's just the disadvantage of a fail-safe system that only leaves a non-functional system in an extreme exceptional situation. Since you haven't provided meaningful logs, I can't say what is going wrong on your end. To rule out an error when applying the overlay, you could start your system with this DTB (rk3588-nanopc-t6.dtb) , which already contains the overlay applied statically. 0 Quote
KingJ Posted yesterday at 09:45 PM Author Posted yesterday at 09:45 PM Thanks for compiling that overlay in to a DTB - i've just tried putting that in to /boot/dtb/rockchip/rk3588-nanopc-t6.dtb, rebooted, and it successfully booted and used HS200 mode for the MMC! [ +0.015056] mmc0: SDHCI controller on fe2e0000.emmc [fe2e0000.emmc] using ADMA [ +0.006109] dwmmc_rockchip fe2c0000.mmc: IDMAC supports 32-bit address mode. [ +0.000031] dwmmc_rockchip fe2c0000.mmc: Using internal DMA controller. [ +0.000008] dwmmc_rockchip fe2c0000.mmc: Version ID is 270a [ +0.000023] dwmmc_rockchip fe2c0000.mmc: DW MMC controller at irq 95,32 bit host data width,256 deep fifo [ +0.000004] mmc0: new HS200 MMC card at address 0001 [ +0.000045] dwmmc_rockchip fe2c0000.mmc: Got CD GPIO [ +0.000236] mmcblk0: mmc0:0001 A3A444 230 GiB [ +0.002488] mmcblk0: p1 [ +0.000267] mmcblk0boot0: mmc0:0001 A3A444 4.00 MiB [ +0.000759] mmcblk0boot1: mmc0:0001 A3A444 4.00 MiB [ +0.000672] mmcblk0rpmb: mmc0:0001 A3A444 4.00 MiB, chardev (243:0) I'm puzzled as to why that worked, but the overlay didn't work. I would have grabbed some logs if I could, but there was no display output so I think the only output would have been on the UART? Unfortunately, I bought the version that comes in a case and the UART is on the top side of the board that I can't access. Searching around, it doesn't look like it is easy to pop it out of the case either. Just in case putting it in to user overlays was wrong, I tried reverting to the original DTB, putting the overlay in /boot/dtb-6.12.44-current-rockchip64/rockchip/overlay/rockchip-rk3588-nanopc-t6-emmc.dtbo, activating it via armbian-config, and rebooting. Unfortunately I hit the same issue as before where it wouldn't boot. Is there a way I can debug why the overlay isn't working without getting access to the UART header on the top of the board? 0 Quote
SuperKali Posted 13 hours ago Posted 13 hours ago @KingJ Which version of the NanoPC T6 do you have? The LTS or non-LTS version? I currently have both, and I will check the eMMC on both. 0 Quote
SuperKali Posted 13 hours ago Posted 13 hours ago ➜ ~ dmesg| grep mmc [ 1.926019] mmc0: SDHCI controller on fe2e0000.mmc [fe2e0000.mmc] using ADMA [ 2.026282] mmc0: new HS400 Enhanced strobe MMC card at address 0001 [ 2.031712] mmcblk0: mmc0:0001 A3A444 230 GiB [ 2.037074] mmcblk0: p1 [ 2.038122] mmcblk0boot0: mmc0:0001 A3A444 4.00 MiB [ 2.040301] mmcblk0boot1: mmc0:0001 A3A444 4.00 MiB [ 2.042564] mmcblk0rpmb: mmc0:0001 A3A444 4.00 MiB, chardev (243:0) [ 2.150020] dwmmc_rockchip fe2c0000.mmc: IDMAC supports 32-bit address mode. [ 2.150727] dwmmc_rockchip fe2c0000.mmc: Using internal DMA controller. [ 2.155343] dwmmc_rockchip fe2c0000.mmc: Version ID is 270a [ 2.155854] dwmmc_rockchip fe2c0000.mmc: DW MMC controller at irq 98,32 bit host data width,256 deep fifo [ 2.156969] dwmmc_rockchip fe2c0000.mmc: Got CD GPIO [ 2.169374] mmc_host mmc1: Bus speed (slot 0) = 400000Hz (slot req 400000Hz, actual 400000HZ div = 0) [ 2.432912] mmc_host mmc1: Bus speed (slot 0) = 198000000Hz (slot req 200000000Hz, actual 198000000HZ div = 0) [ 2.714811] dwmmc_rockchip fe2c0000.mmc: Successfully tuned phase to 230 [ 2.717703] mmc1: new UHS-I speed SDR104 SDXC card at address 544c [ 2.719268] mmcblk1: mmc1:544c LX64G 58.9 GiB [ 2.721418] mmcblk1: p1 p2 [ 5.394333] EXT4-fs (mmcblk1p2): mounted filesystem 00560a2f-4a48-4f42-85db-f7197f411193 ro with ordered data mode. Quota mode: none. [ 8.122124] EXT4-fs (mmcblk1p2): re-mounted 00560a2f-4a48-4f42-85db-f7197f411193 r/w. Quota mode: none. ➜ ~ uname -a Linux t6.superkali.lan 6.14.4-edge-rockchip64 #1 SMP PREEMPT Fri Apr 25 08:51:21 UTC 2025 aarch64 GNU/Linux root@t6-2:~# dmesg | grep mmc [ 1.793233] mmc0: SDHCI controller on fe2e0000.mmc [fe2e0000.mmc] using ADMA [ 1.892860] mmc0: new HS400 Enhanced strobe MMC card at address 0001 [ 1.893418] mmcblk0: mmc0:0001 A3A444 230 GiB [ 1.900752] mmcblk0: p1 [ 1.901187] mmcblk0boot0: mmc0:0001 A3A444 4.00 MiB [ 1.902094] mmcblk0boot1: mmc0:0001 A3A444 4.00 MiB [ 1.903152] mmcblk0rpmb: mmc0:0001 A3A444 4.00 MiB, chardev (243:0) [ 2.015519] dwmmc_rockchip fe2c0000.mmc: IDMAC supports 32-bit address mode. [ 2.015588] dwmmc_rockchip fe2c0000.mmc: Using internal DMA controller. [ 2.015603] dwmmc_rockchip fe2c0000.mmc: Version ID is 270a [ 2.015721] dwmmc_rockchip fe2c0000.mmc: DW MMC controller at irq 98,32 bit host data width,256 deep fifo [ 2.016264] dwmmc_rockchip fe2c0000.mmc: Got CD GPIO [ 2.029321] mmc_host mmc1: Bus speed (slot 0) = 400000Hz (slot req 400000Hz, actual 400000HZ div = 0) [ 2.270630] mmc_host mmc1: Bus speed (slot 0) = 198000000Hz (slot req 200000000Hz, actual 198000000HZ div = 0) [ 2.473628] dwmmc_rockchip fe2c0000.mmc: Successfully tuned phase to 247 [ 2.477145] mmc1: new UHS-I speed SDR104 SDXC card at address 544c [ 2.478144] mmcblk1: mmc1:544c LX64G 58.9 GiB [ 2.483356] mmcblk1: p1 [ 3.176039] EXT4-fs (mmcblk1p1): mounted filesystem 3aa41aca-1f77-4222-b253-8d444c3dfca5 ro with writeback data mode. Quota mode: none. [ 8.484634] EXT4-fs (mmcblk1p1): re-mounted 3aa41aca-1f77-4222-b253-8d444c3dfca5 r/w. Quota mode: none. root@t6-2:~# uname -a Linux t6-2.superkali.lan 6.14.4-edge-rockchip64 #1 SMP PREEMPT Fri Apr 25 08:51:21 UTC 2025 aarch64 GNU/Linux root@nanopct6-lts:~# dmesg | grep mmc [ 4.386835] dwmmc_rockchip fe2c0000.mmc: No normal pinctrl state [ 4.386849] dwmmc_rockchip fe2c0000.mmc: No idle pinctrl state [ 4.386921] dwmmc_rockchip fe2c0000.mmc: IDMAC supports 32-bit address mode. [ 4.386972] dwmmc_rockchip fe2c0000.mmc: Using internal DMA controller. [ 4.386986] dwmmc_rockchip fe2c0000.mmc: Version ID is 270a [ 4.387034] dwmmc_rockchip fe2c0000.mmc: DW MMC controller at irq 157,32 bit host data width,256 deep fifo [ 4.387131] dwmmc_rockchip fe2c0000.mmc: Looking up vmmc-supply from device tree [ 4.387431] dwmmc_rockchip fe2c0000.mmc: Looking up vqmmc-supply from device tree [ 4.403026] mmc_host mmc1: Bus speed (slot 0) = 400000Hz (slot req 400000Hz, actual 400000HZ div = 0) [ 4.495259] mmc0: Command Queue Engine enabled [ 4.495299] mmc0: new HS400 Enhanced strobe MMC card at address 0001 [ 4.496710] mmcblk0: mmc0:0001 A3A561 57.6 GiB [ 4.505219] mmcblk0: p1 [ 4.505766] mmcblk0boot0: mmc0:0001 A3A561 4.00 MiB [ 4.506882] mmcblk0boot1: mmc0:0001 A3A561 4.00 MiB [ 4.507923] mmcblk0rpmb: mmc0:0001 A3A561 16.0 MiB, chardev (234:0) root@nanopct6-lts:~# uname -a Linux nanopct6-lts 6.1.84-vendor-rk35xx #1 SMP Wed Dec 18 05:56:40 UTC 2024 aarch64 GNU/Linux root@nanopct6-lts:~# The first two are the variants with 256GB of eMMC, therefore non-LTS, while the last one is the LTS variant with 64GB of eMMC. From my tests with both the edge version and the vendor kernel, both negotiate in HS400 without any problems: A test carried out by T6-2: /dev/mmcblk0p1: Timing cached reads: 7686 MB in 2.00 seconds = 3845.97 MB/sec Timing buffered disk reads: 854 MB in 3.00 seconds = 284.38 MB/sec I don't see your errors, so I think there might be something strange with your board. Can you try the edge kernel and tell me if you have the same problems, or the vendor kernel? 0 Quote
usual user Posted 2 hours ago Posted 2 hours ago 21 hours ago, KingJ said: Thanks for compiling that overlay in to a DTB That was just a simple one-liner (fdtoverlay -i rk3588-nanopc-t6.dtb -o rk3588-nanopc-t6.dtb rk3588-nanopc-t6-emmc.dtbo). In order to carry out the test in my environment, the typing task was a little more complex: Spoiler Executed: fdtoverlay --input rk3588-nanopc-t6-lts-native.dtb --output rk3588-nanopc-t6-lts-con-btn-npu-emmc.dtb rk3588-nanopc-t6-lts-con.dtbo rk3588-nanopc-t6-lts-btn.dtbo rk3588-nanopc-t6-lts-npu.dtbo rk3588-nanopc-t6-lts-emmc.dtbo Extended extlinux.conf with this boot stanza: label rockchip menu label ODROID-M1/M2 NanoPC-T6 emmc fdt /usr/lib/modules/linux/dtb/rockchip/rk3588-nanopc-t6-lts-con-btn-npu-emmc.dtb kernel /usr/lib/modules/linux/vmlinuz append loglevel=4 root=PARTUUID=6a6d7001-01 coherent_pool=2M selinux=0 audit=0 console=ttyS02,1500000 console=tty0 fbcon=nodefer rootwait rootfstype=ext4 raid=noautode Because my device that provides the serial terminal was just shut down for maintenance purposes, I set it as the default boot option. In this way, I can delay the start up of the serial terminal to that time I am faced with a unbootable system. Unfortunately, everything worked and I didn't have to reboot with a fail-safe option. And I also didn't accidentally type a configuration incorrectly that would have triggered an automatic safety start. 21 hours ago, KingJ said: I'm puzzled as to why that worked, but the overlay didn't work. It apparently depends on how the overlay was applied, because fundamentally it should also work when applied dynamically. However, the static application of an overlay has the disadvantage that in the case of an incompatibility, one is only confronted with an error message and does not experience a system that fails to start. 21 hours ago, KingJ said: Searching around, it doesn't look like it is easy to pop it out of the case either. It is not very difficult to get it out. It is held only by the slight adhesive strength of the thermal pad between the SoC and the casing when the bottom plate is removed. A cautious light steady pull releases it. The only difficulty is in gripping the board to make apply this pull. I screwed a bolt into one of the mounting PCB nuts and used it as a handle. With such a socket in one of the SMA antenna connection ports, the UART connection can be permanently routed to the outside without modifying the casing. I use it to route the fan connector outside. 22 hours ago, KingJ said: Is there a way I can debug why the overlay isn't working without getting access to the UART header on the top of the board? Without the possibility of providing meaningful serial console logs, you will probably be left on your own with it. 0 Quote
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.