Banana Pi M2 Ultra boot freezing trying to scan the EMMC

[goosnarrggh]

I've been trying to bring up Armbian on my Banana Pi M2 Ultra (with A40i SoC).

 

First I tried using the community rolling release, and then building it myself from source code. In every case, it was failing to boot; on the HDMI console it was hanging at the stage where the initrd was attempting to scan for the root partition. ("Scanning for btrfs filesystems")

 

Taking a closer look at the verbose output of the UART console during a boot attempt, I saw that there was an error message about the sunxi-mmc device driver experiencing a failure while it was enumerating the EMMC chip:

[    4.136553] mmcblk2: mmc2:0001 8GTF4R 7.28 GiB
[    4.144362] sunxi-mmc 1c11000.mmc: data error, sending stop command
[    4.150727] sunxi-mmc 1c11000.mmc: send stop command failed

 

On a hunch, I patched the device tree to disable that interface. After compiling that change and installing a new image on a microSD card, the board booted flawlessly, and I am now in a working desktop environment.

 

This leads me to a question: How likely is it that there's just some damaged/defect on my particular board which makes its EMMC unusable? (In which case I might stop here, and be satisfied with the result: Because, I'm not particularly interested in working with the EMMC chip anyway.)

 

Or, it is possible that there's a systemic problem with the Banana Pi M2 Ultra device tree which would be equally impacting everyone's ability to boot? (In which case it might be worthwhile to dig a little deeper, try to figure out why the kernel is hanging when it attempts to enumerate the EMMC chip, and perhaps submit a patch to fix the problem.)

 

Has anyone else experienced this? Or, in case this is a systemic problem, does anyone have some advice about the most appropriate next steps to contribute a fix back to the community?

[goosnarrggh]

Follow-up: Given the fact that Linux was able to detect the presence of the eMMC chip at all, and that it was able to correctly identify the capacity, I figured that it was at least able to get rudimentary communication up and running. Since every MMC always starts up in SDR 1-bit mode by default, and then the host has the option to interrogate the chip for the possibility to negotiate a higher bus width and/or clock mode, I figured I'd try carefully re-introducing the interface with more limited capabilities.

 

My board has a Samsung KLM8G1GETF eMMC chip, and that part supports single-data-rate 26 MHz at all possible I/O voltages, single-data-rate 52 MHz at all possible I/O voltages, double-data-rate 52 MHz at all possible I/O voltages, HS200 (single-data-rate 200 Mhz at 1.8V only). and HS400 (double-data-rate 200 Mhz at 1.8V only).

 

Looking at the voltage rail fan-out in the schematic, it's pretty clear that it's physically impossible to power the eMMC's I/O bus at anything other than 3.3V, and so HS200 and HS400 are definitively off the table.

 

But the device tree bindings for sunxi-mmc don't seem to provide very much configuration to take away modes which I know cannot be supported. Really just the bus width. So, since it was apparent that Linux must have successfully connected to the eMMC in a 1-bit mode before, I tried re-enabling the eMMC interface, but limiting its bus width to 1 bit. At least that would eliminate the unsupportable HS200 and HS400 as possibilities; they cannot operate on a data bus narrower than 4 bits.

 

It worked, Armbian booted successfully, and I am able to see the eMMC and manipulate its partitions. It also seems to have detected the device as high-speed (and presumably single-data-rate):

[    2.481613] mmc2: new high speed MMC card at address 0001
[    2.483323] mmcblk2: mmc2:0001 8GTF4R 7.28 GiB
[    2.488509] mmcblk2boot0: mmc2:0001 8GTF4R 4.00 MiB
[    2.492442] mmcblk2boot1: mmc2:0001 8GTF4R 4.00 MiB

 

This is a reasonable milestone, I suppose -- the chip works and (not pictured) I can create and format partitions.

 

But it's still not ideal. It is supposed to be possible to operate with:

- SDR 4-bit and 8-bit data bus

- DDR at 52 MHz with 1-bit, 4-bit, and 8-bit data bus.

 

Ideally, I would like to try out each of these combinations in isolation so I can figure out where things are breaking. But it looks like this might be the sort of thing that I will not be able to configure using a patched dts alone - I might need to actually dig into the kernel sources themselves.

 

I still would appreciate pointers about where to go next.

Edited May 9, 2023 by goosnarrggh
fix up dmesg excerpt

[joximus]

Since the board is unusable with existing image it would be nice to see your changes implemented in some of the future images. Or could you write some howto for beginners?

Edited May 10, 2023 by joximus

[goosnarrggh]

I have a suspicion that the problem with the current image is probably limited to rev 1.1 boards based on the A40i SoC - it's entirely possible that the older rev 1 boards with an R40 SoC aren't having any problems at all.

If that's the case, then the changes I have proposed should be backward compatible with both older and newer boards, however it would come at the expense of a significant eMMC slowdown for users of the older boards. I'm not sure if the project admins would be willing to accept a pull request which would involve that level of compromise.

 

But maybe a little howto describing the minimum set of changes someone would need to make in order to patch the existing image to make it work for them might be in order.

 

Right now, I'm looking into the possibility of making this configuration change into a dtbo which someone could just drop onto a stock image.

 

By the same token, a PR could also convert the stock image to a 1-bit bus, and an optional dtbo could be supplied to allow people to increase the bus back to 8-bits if they know their hardware is capable of it.

[goosnarrggh]

Update: Yes, a dtbo will work to reduce the bus from 8-bits to 1-bit and make the board bootable.

 

This overlay does the trick for me:

sun8i-r40-bananapi-m2-ultra-1bit-emmc.dtssun8i-r40-bananapi-m2-ultra-1bit-emmc.dtbo

 

To test this for yourself: (You will need a working Linux machine to mount the SD card and make these changes.)

1. Burn a copy of the current rolling release image onto a microSD card.
2. Copy the dtbo file into the /boot/overlay-user/ subdirectory of the card. (You will need to use root/superuser access to have permission to modify the content of this directory on the SD card. Create that subdirectory if it doesn't already exist.)
3. Edit the file /boot/armbianEnv.txt. (You will need to have root/superuser access to have permission to edit this file on the SD card.)
   1. If there's no line starting with "user_overlays=" in the file, then just add a new line to the end of the file:

      user_overlays= sun8i-r40-bananapi-m2-ultra-1bit-emmc

   2. Otherwise, if a line starting with "user_overlays=" is already present in the, then add a space to the end of that line's existing contents, and add sun8i-r40-bananapi-m2-ultra-1bit-emmc to the end of it.

 

Note: I linked both the dtbo binary, and the dts source code which created it. Only the dtbo file is needed for a quick test; if you wish you could compile the dts file from source code and prove that my work is reproducible.

Edited May 11, 2023 by goosnarrggh
Added explanation that the .dts file is supplied for reference only.

[joximus]

Thank you very much. I will try it tomorrow and report results here.

[joximus]

I tried quick test with dtbo file you provided and it works. This is the first time I have this board up and running. It would be great if you changes could somehow be included in future images provided for download.

[jamess4451]

Thank-you -- I can confirm this fixes the issue and allows the board to finally boot in something other than their provided kernel version 3.3.

[goosnarrggh]

Unfortunately, on another forum some other people have reported failure when they tried using my proposed fix. So although it's working for some people, it doesn't seem to be sufficient to fix everyone's problems. I'm hesitant to proceed with a pull request armed with these mixed results.

[Super8film]

any progress? so I tried it by myself but it is not working

Edited May 27, 2023 by Super8film

[goosnarrggh]

I'm afraid I have taken it as far as I can with the equipment I have: My own hardware is now working fine, and whatever problems other people are having must be due to a different root cause. Since I am not having that problem (whatever it is), there's no way for me to even begin to try to fix it.

 

@Super8film do you have a USB-TTL serial cable? If so, then maybe you could hook it up (serial terminal settings for this board are 8N1-57600 bps) and try to capture a verbose log of what the bootloader and kernel init process have to say while the board is booting? That might provide some evidence that could be used to try to solve your problem too.

[Super8film]

Thx I'll try this once I've time. Thx

 

[Miriam Maerx]

This is amazing, thank you goosnarrggh!
 

The dtbo-file works for me exactly as described. My armbian is booting again and you saved me from completly giving up on the bananapi.

 

If it helps at all to narrow down where the issue comes from: The current (and some earlier) minimal image of armbian worked for me initally, but after installing and updating some packages (including a rebuild of initramfs), I consistently ran into the freeze at "Scanning for btrfs filesystems".

Edited October 8, 2023 by Miriam Maerx
typo

[jimw]

Nice. Thank you.