Not booting from emmc after moving sd -> nand-install tool

[Franky66]

Hi there, yesterday I tried to use nand install skript tool to move the content from my sd card on odroid n2 to emmc card for booting and using emmc card only. The process itself worked and the content of the sd cards was moved, but after unplugging sd card and plugging emmc card the boot process is not starting and nothing is shown on the hdmi monitor. I compared the /boot filesystems and both are identically. I checked the "armbianEnv.txt" file and it is ok for the new uuid of the emmc card.

 

Everything seems to be identically but why is it not booting. Does someone have a hint what to check additionally what is maybe missing at beginning of the boot process. The LED is not blinking on boot process so something in the first steps of booting and loading does not work.

 

Thank you in advance

[Franky66]

Aditionally my actual configuration

- upgraded armbian debian 10 build with omv5 installation to armbian debian 11 build with omv6 via omv-upgrade-install successfull

- actual kernel running 5.10.102-meson64 #22.02.1 SMP PREEMPT

- output from /boot directoy

 

drwxr-xr-x 10 root root     4096 May 19 11:27 .
drwxr-xr-x 21 root root     4096 May 17 21:31 ..
-rw-r--r--  1 root root        0 May 13 08:54 .next
lrwxrwxrwx  1 root root       24 May 13 08:54 Image -> vmlinuz-5.10.102-meson64
-rw-r--r--  1 root root  5292029 Feb 27 10:09 System.map-5.10.102-meson64
-rw-r--r--  1 root root      180 May 19 11:27 armbianEnv.txt
-rw-r--r--  1 root root     1536 Mar 21  2020 armbian_first_run.txt.template
-rw-r--r--  1 root root     4882 Mar 21  2020 boot-desktop.png
-rw-r--r--  1 root root    38518 Mar 21  2020 boot.bmp
-rw-r--r--  1 root root     7823 May 13 08:29 boot.cmd
-rw-r--r--  1 root root     5579 May 13 08:29 boot.ini
-rw-r--r--  1 root root     7895 May 13 08:29 boot.scr
-rw-r--r--  1 root root   229458 Feb 27 10:09 config-5.10.102-meson64
-rwxr-xr-x  1 root root      120 Jan  1  1970 display.bin
lrwxrwxrwx  1 root root       20 May 13 08:54 dtb -> dtb-5.10.102-meson64
drwxr-xr-x  2 root root     4096 Aug 23  2020 dtb-4.9.219-meson64
drwxr-xr-x  2 root root     4096 Sep 21  2020 dtb-4.9.232-meson64
drwxr-xr-x  2 root root     4096 Oct 27  2020 dtb-4.9.236-meson64
drwxr-xr-x  2 root root     4096 Feb  7  2021 dtb-4.9.247-meson64
drwxr-xr-x  2 root root     4096 Mar 10  2021 dtb-4.9.250-meson64
drwxr-xr-x  2 root root     4096 May 12  2021 dtb-4.9.260-meson64
drwxr-xr-x  2 root root     4096 May 13 08:53 dtb-4.9.277-meson64
drwxr-xr-x  4 root root     4096 May 13 08:53 dtb-5.10.102-meson64
lrwxrwxrwx  1 root root       19 Aug 31  2021 dtb.old -> dtb-4.9.275-meson64
-rwxr-xr-x  1 root root      256 Jan  1  1970 edid.bin
-rw-r--r--  1 root root 16773114 May 17 21:53 initrd.img-5.10.102-meson64
-rw-r--r--  1 root root 26329664 May 13 08:54 uImage
lrwxrwxrwx  1 root root       24 May 17 21:53 uInitrd -> uInitrd-5.10.102-meson64
-rw-r--r--  1 root root 16773178 May 17 21:53 uInitrd-5.10.102-meson64
-rw-r--r--  1 root root 26329600 Feb 27 10:09 vmlinuz-5.10.102-meson64
lrwxrwxrwx  1 root root       23 May 13 08:30 zImage -> vmlinuz-4.9.277-meson64

 

[usual user]

On 5/19/2022 at 10:47 AM, Franky66 said:

Does someone have a hint what to check additionally what is maybe missing at beginning of the boot process.

What firmware (uboot) version are you using and where is it located (SPI or eMMC)?

[Franky66]

vor 23 Stunden schrieb usual user:

What firmware (uboot) version are you using and where is it located (SPI or eMMC)?

How to find out the uboot version? I am not using SPI as it is not working at the moment. Don't know why, flashed the latest version and if sd card is inserted it crashes directly to shell if I press a button on keyboard.

 

From package manager I get the following information:

 

ii  linux-u-boot-odroidn2-current     22.02.1                        arm64        Uboot loader 2022.01
ii  u-boot-tools                      2021.01+dfsg-5                 arm64        companion tools for Das U-Boot bootloader

[usual user]

4 hours ago, Franky66 said:

How to find out the uboot version?

Version information is output to the serial console as well as other helpful information. It is therefore a good idea to post the entire log.
I use my self-build firmware (2022.01) in the SPI flash. Since I don't have an eMMC available at the moment, I can't say if this version basically works with eMMC. I just built the 2022.07-rc2. If you want to experiment with it, I can upload the firmware and you can  try it with a spare SD card. Maybe some improvements have already been incorporated there.

[Franky66]

I am sorry, I do not have the separate reader to get the serial console information actually. I will also try to find some support in the hardkernel forum how to get things booting. Maybe something else is broken as the updating petiboot via recovery image didn't work at first time and I had to zero the internal flash areas and try again. There was some advise for this in the forum

[usual user]

21 minutes ago, Franky66 said:

Maybe something else is broken as the updating petiboot via recovery image didn't work

Petiboot can't cope with the Armbian distribution, the reason why I replaced it by mainline uboot.

[Franky66]

Ok so is there a reason why they conflict each other. I can try you self-build firmware with only u-boot and flash to spi. I have one odroid n2 for experimenting and testing.

[usual user]

On 5/22/2022 at 9:20 AM, Franky66 said:

is there a reason why they conflict each other

The boot flow they implemented is simply incompatible with Armbian boot method. And since it is also closed source, where I would not even have the opportunity to repair it, for me it is reason to reject it unconditionally.

 

On 5/22/2022 at 9:20 AM, Franky66 said:

I can try you self-build firmware

Since you don't have access to the serial console, a basic test only requires an SD card that you can delete completely. An installation in SPI flash or eMMC only makes sense if it is previously ensured that the firmware basically works for your use case. Debugging an installation in SPI flash or eMMC without access to log files is hopeless. If the firmware basically works, you can install it in SPI flash or eMMC in a next step.

 

These are the necessary steps for the experiment:
Prepare a completely blank SD card, e.g.:

Quote

dd if=/dev/zero of=/dev/${entire-SD-card-device-to-be-cleared}

Copy my firmware binary to the empty SD card, e.g.:

Quote

dd bs=512 seek=1 conv=notrunc if=u-boot-meson.bin of=/dev/${entire-SD-card-device-to-be-used} ; sync

Now insert the SD card into the Odroid N2, mount the eMMC module with the system to be booted, switch the boot switch to the MMC position and start the Odroid N2.
If the system boots successfully now, you can think about installing the firmware in the SPI flash or the eMMC.
When you're ready to do the experiment, speak up and I'll upload the firmware.

[ManoftheSea]

12 hours ago, usual user said:

Prepare a completely blank SD card

You might try "blkdiscard" rather than a dd of zeros.  It ought to be faster, and will have better performance afterward besides.

[usual user]

2 hours ago, ManoftheSea said:

It ought to be faster, and will have better performance afterward besides.

I asked for a nulled SD card. Since you did not mention the "-k" parameter of blkdiscard in your proposal, this is not guaranteed. Firmware does not take into account filesystem structures and accesses storage media raw, so remaining data fragments can lead to disturbances. Since the write function of the storage medium accounts for the majority of the operation, the execution time of both programs should be almost identical.
The use of an additional tool also requires appropriate knowledge to use it. This increases the complexity of the activity to be carried out. The use of the same tool always reduces the effort to explain the use of the tools used, you can build on already imparted knowledge.

[Franky66]

hello, I have prepared a sd card as described and "nulled" it via dd. So I can flash you firmware on spi for testing.

[usual user]

I uploaded an archive of my firmware here. Unpack it with

tar -xzf u-boot-meson.tgz

and use the dd command described above to write it to the prepared SD card.
And then keep your fingers crossed, hopefully it works for your use case.

[ManoftheSea]

20 hours ago, usual user said:

"-k" parameter of blkdiscard

blkdiscard doesn't have a "-k" parameter in my distro.  What does yours do?.  By default, blkdiscard tells the sd card to zero out all memory and prepare it all for writing.  On the sd card, it is not the "write" operation that takes the most time, but the erase operation. 
blkdiscard: tells the card "mark everything available and prepare it whenever you have a chance", making the pool for writes the maximum.
dd: tells the card that every sector is written to, shrinking the spare pool of sectors to the minimum.

It is worth knowing the difference between "write every sector (to zero)" and "tell the card to empty itself".

[usual user]

59 minutes ago, ManoftheSea said:

What does yours do?

The man page of my version corresponds with many hits of a google search of "man blkdiscard", e.g.:https://www.man7.org/linux/man-pages/man8/blkdiscard.8.html
And looks like I fat finger typed the parameter, I meant -z, --zeroout.

[Franky66]

So I wrote your image to the sd card. Both sd card and emmc are plugged in, but after power on the screen stays black, nothing is shown. I need this serial to usb adapter, that's useless in finding bugs. Time to buy :-(

[usual user]

10 hours ago, Franky66 said:

but after power on the screen stays black

Even without an eMMC mounted, you should see the following messages:

Spoiler

Board variant: n2-plus
Net:   eth0: ethernet@ff3f0000
starting USB...
Bus usb@ff500000: Register 3000140 NbrPorts 3
Starting the controller
USB XHCI 1.10
scanning bus usb@ff500000 for devices... 5 USB Device(s) found
       scanning usb for storage devices... 0 Storage Device(s) found
Hit any key to stop autoboot:  0
switch to partitions #0, OK
mmc0 is current device
Card did not respond to voltage select! : -110
MMC Device 2 not found
no mmc device at slot 2

Device 0: unknown device
ethernet@ff3f0000 Waiting for PHY auto negotiation to complete......... TIMEOUT !
Could not initialize PHY ethernet@ff3f0000
missing environment variable: pxeuuid
Retrieving file: pxelinux.cfg/01-00-1e-06-42-ce-b9
ethernet@ff3f0000 Waiting for PHY auto negotiation to complete......... TIMEOUT !
Could not initialize PHY ethernet@ff3f0000
Retrieving file: pxelinux.cfg/00000000

 

And you are sure your boot mode switch is in the MMC position (towards the SD card slot)?
Maybe something went wrong when preparing the SD card.
This is what the messages look like when I write the SD card:

Spoiler

dd bs=512 seek=1 conv=notrunc if=u-boot-meson.bin of=/dev/sdc ; sync
2097+1 records in
2097+1 records out
1074032 bytes (1,1 MB, 1,0 MiB) copied, 0,381472 s, 2,8 MB/s

 

Please be able to read back your prepared SD card with the following command:
 

dd count=4096 if=/dev/${entire-SD-card-device-prepared} of=u-boot-meson-readback.bin

If you upload the u-boot-meson-readback.bin, I can take a look and check if everything is ok. You may also post what your used command lines look like.

[ManoftheSea]

20 hours ago, usual user said:

I meant -z, --zeroout

Alright, I agree, including "-z" to blkdiscard is better than using dd to mark every block as used.  It appears in the best case, the device will make a promise to return zeros instead of garbage data, and in the worst case, Linux will write all the blocks itself.

 

Franky66, it may be worthwhile to try with just the SD with the Usual User Firmware, without the eMMC, and see if you get any output.  Then add the eMMC, and see what changes.

[usual user]

1 hour ago, ManoftheSea said:

without the eMMC, and see if you get any output

My shown messages are emitted by the firmware which should have been loaded from the maskrom code from the firware area of the SD card. Since no other drives have been considered until then, it is irrelevant whether an eMMC is mounted or not. If no messages are displayed, this indicates that no firmware is running. Either the maskrom code cannot read the SD card, or the firmware is not in the required positions. In order to check the positioning, I asked for the dump of the first 2MB of the SD card. A serial console log does not provide any significant new insights at this time.

[Franky66]

vor 23 Stunden schrieb usual user:

My shown messages are emitted by the firmware which should have been loaded from the maskrom code from the firware area of the SD card. Since no other drives have been considered until then, it is irrelevant whether an eMMC is mounted or not. If no messages are displayed, this indicates that no firmware is running. Either the maskrom code cannot read the SD card, or the firmware is not in the required positions. In order to check the positioning, I asked for the dump of the first 2MB of the SD card. A serial console log does not provide any significant new insights at this time.

Thankx for your help, I am some days on holiday and will respond as soon as possible.

[Franky66]

Am 24.5.2022 um 20:35 schrieb usual user:

My shown messages are emitted by the firmware which should have been loaded from the maskrom code from the firware area of the SD card. Since no other drives have been considered until then, it is irrelevant whether an eMMC is mounted or not. If no messages are displayed, this indicates that no firmware is running. Either the maskrom code cannot read the SD card, or the firmware is not in the required positions. In order to check the positioning, I asked for the dump of the first 2MB of the SD card. A serial console log does not provide any significant new insights at this time.

Hello, I did flashing again sd card without any changes:

 

Spoiler

root@n2-db-omv5-test:~# dd bs=512 seek=1 conv=notrunc if=u-boot-meson.bin of=/dev/sdg ; sync
2097+1 records in
2097+1 records out
1074032 bytes (1.1 MB, 1.0 MiB) copied, 0.342007 s, 3.1 MB/s

And for comparing I read the sd card back to file

Spoiler

root@n2-db-omv5-test:~# dd count=4096 if=/dev/sdg of=u-boot-meson-readback.bin
4096+0 records in
4096+0 records out
2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.181937 s, 11.5 MB/s

Attached please find the corresponding readback file.

u-boot-meson-readback.bin.gz

[usual user]

1 hour ago, Franky66 said:

Attached please find the corresponding readback file.

I have checked the dump, and everything is as it should be. Looks like the maskrom code can't read the SD card, I've experienced this with SD cards that weren't at least UHS-1 class. So, for example, class 4, class 10, ...
What comes to my mind right now, what native resolution does your display have? If it is not a standard HDMI resolution, it could be that it is not supported by the firmware. The startup process should still continue, but may hang later for another reason. And without serial console outputs, you're relatively blind.
To rule out the SD card as the culprit, you can also write the firmware to the eMMC. It is the same command as for the SD card:

dd bs=512 seek=1 conv=notrunc if=u-boot-meson.bin of=/dev/${entire-eMMC-device-to-be-used} ; sync

However, you should only do this if you have a backup of the eMMC in order not to lose any data. Whether the OS can finally be started is uncertain, but the first start messages should be displayed if there were SD card reading problems. The boot attempt should only be carried out with mounted eMMC and without SD card in the SD card slot. The start mode switch should be in the MMC position to exclude any SPI flash involvement.
If all this does not lead to a result, access to the serial console is essential to continue troubleshooting.

[MichaIng]

@Franky66 did you manage to boot from eMMC when flashing it directly there? I'm not able to get the recent Odroid N2 Bullseye image to boot from eMMC at all, regardless which way. Doing the exactly same on SD card works fine. I also tried it with a custom image build, installing the Armbian "current" kernel package (Linux 5.10.123) and flashing respective U-Boot via write_uboot_platform: Fails the same way when flashing to eMMC, works from SD card. Here is the log from serial console:

 

Spoiler

G12B:BL:6e7c85:2a3b91;FEAT:E0F83180:402000;POC:F;RCY:0;EMMC:0;READ:0;0.
bl2_stage_init 0x01
bl2_stage_init 0x81
hw id: 0x0000 - pwm id 0x01
bl2_stage_init 0xc1
bl2_stage_init 0x02

L0:00000000
L1:00000703
L2:0000c067
L3:14000020
B2:00402000
B1:e0f83180

TE: 142147

BL2 Built : 06:17:13, Jun 28 2019. g12b gf0505d7-dirty - qi.duan@droid13

Board ID = 5
Set A53 clk to 24M
Set A73 clk to 24M
Set clk81 to 24M
A53 clk: 1200 MHz
A73 clk: 1200 MHz
CLK81: 166.6M
smccc: 00027377
eMMC boot @ 0
sw8 s
DDR driver_vesion: LPDDR4_PHY_V_0_1_14 build time: Jun 28 2019 06:17:09
board id: 5
Load FIP HDR from eMMC, src: 0x00010200, des: 0xfffd0000, size: 0x00004000, part: 0
fw parse done
Load ddrfw from eMMC, src: 0x00060200, des: 0xfffd0000, size: 0x0000c000, part: 0
Load ddrfw from eMMC, src: 0x00038200, des: 0xfffd0000, size: 0x00004000, part: 0
PIEI prepare done
fastboot data load
00000000
emmc switch 1 ok
00000000
emmc switch 2 ok
fastboot data verify
verify result: 255
Cfg max: 2, cur: 1. Board id: 255. Force loop cfg
DDR4 probe
ddr clk to 1320MHz
Load ddrfw from eMMC, src: 0x00014200, des: 0xfffd0000, size: 0x0000c000, part: 0
00000000
emmc switch 0 ok
Check phy result
INFO : End of initialization
INFO : End of read enable training
INFO : End of fine write leveling
INFO : End of read dq deskew training
INFO : End of MPR read delay center optimization
INFO : End of Write leveling coarse delay
INFO : End of write delay center optimization
INFO : End of read delay center optimization
INFO : End of max read latency training
INFO : Training has run successfully!
1D training succeed
Load ddrfw from eMMC, src: 0x00020200, des: 0xfffd0000, size: 0x0000c000, part: 0
Check phy result
INFO : End of initialization
INFO : End of 2D read delay Voltage center optimization
INFO : End of 2D write delay Voltage center optimization
INFO : Training has run successfully!

R0_RxClkDly_Margin==94 ps 8
R0_TxDqDly_Margi==118 ps 10


R1_RxClkDly_Margin==0 ps 0
R1_TxDqDly_Margi==0 ps 0

 dwc_ddrphy_apb_wr((0<<20)|(2<<16)|(0<<12)|(0xb0):0001
2D training succeed
auto size-- 65535DDR cs0 size: 2048MB
DDR cs1 size: 2048MB
DMC_DDR_CTRL: 00600024DDR size: 3928MB
cs0 DataBus test pass
cs1 DataBus test pass
cs0 AddrBus test pass
cs1 AddrBus test pass
 pre test  bdlr_100_average==435 bdlr_100_min==435 bdlr_100_max==435 bdlr_100_cur==435
 aft test  bdlr_100_average==435 bdlr_100_min==435 bdlr_100_max==435 bdlr_100_cur==435
non-sec scramble use zero key
ddr scramble enabled

100bdlr_step_size ps== 435
result report
boot times 0Enable ddr reg access
00000000
emmc switch 3 ok
Authentication key not yet programmed
get rpmb counter error 0x00000007
00000000
emmc switch 0 ok
Load FIP HDR from eMMC, src: 0x00010200, des: 0x01700000, size: 0x00004000, part: 0
Load BL3X from eMMC, src: 0x0006c200, des: 0x0175c000, size: 0x00096200, part: 0
0.0;M3 CHK:0;cm4_sp_mode 0
E30HDR
MVN_1=0x00000000
MVN_2=0x00000000
[Image: g12b_v1.1.3375-8f9c8a7 2019-01-24 10:44:46 guotai.shen@droid11-sz]
OPS=0x40
ring efuse init
chipver efuse init
29 0c 40 00 01 17 16 00 00 03 32 33 32 58 33 50
[0.019859 Inits done]
secure task start!
high task start!
low task start!
run into bl31
NOTICE:  BL31: v1.3(release):ab8811b
NOTICE:  BL31: Built : 15:03:31, Feb 12 2019
NOTICE:  BL31: G12A normal boot!
NOTICE:  BL31: BL33 decompress pass
ERROR:   Error initializing runtime service opteed_fast


U-Boot 2022.01-armbian (Jun 22 2022 - 07:16:34 +0000) odroid-n2/n2-plus

Model: Hardkernel ODROID-N2
SoC:   Amlogic Meson G12B (S922X) Revision 29:c (40:2)
DRAM:  3.8 GiB
MMC:   sd@ffe05000: 0, mmc@ffe07000: 1
Loading Environment from nowhere... OK
In:    serial
Out:   serial
Err:   serial
Board variant: n2-plus
Net:   eth0: ethernet@ff3f0000
Hit any key to stop autoboot:  0
Card did not respond to voltage select! : -110
switch to partitions #0, OK
mmc1(part 0) is current device
** No partition table - mmc 1 **
Couldn't find partition mmc 1:1
MMC Device 2 not found
no mmc device at slot 2
starting USB...
Bus usb@ff500000: Register 3000140 NbrPorts 3
Starting the controller
USB XHCI 1.10
scanning bus usb@ff500000 for devices... 4 USB Device(s) found
       scanning usb for storage devices... 0 Storage Device(s) found

Device 0: unknown device
ethernet@ff3f0000 Waiting for PHY auto negotiation to complete......... TIMEOUT !
Could not initialize PHY ethernet@ff3f0000
missing environment variable: pxeuuid
Retrieving file: pxelinux.cfg/01-00-1e-06-42-fb-bb
ethernet@ff3f0000 Waiting for PHY auto negotiation to complete......... TIMEOUT !
Could not initialize PHY ethernet@ff3f0000
Retrieving file: pxelinux.cfg/00000000
ethernet@ff3f0000 Waiting for PHY auto negotiation to complete......... TIMEOUT !
Could not initialize PHY ethernet@ff3f0000
Retrieving file: pxelinux.cfg/0000000
ethernet@ff3f0000 Waiting for PHY auto negotiation to complete......... TIMEOUT !
Could not initialize PHY ethernet@ff3f0000
Retrieving file: pxelinux.cfg/000000
ethernet@ff3f0000 Waiting for PHY auto negotiation to complete......... TIMEOUT !
Could not initialize PHY ethernet@ff3f0000
Retrieving file: pxelinux.cfg/00000
ethernet@ff3f0000 Waiting for PHY auto negotiation to complete......... TIMEOUT !
Could not initialize PHY ethernet@ff3f0000
Retrieving file: pxelinux.cfg/0000
ethernet@ff3f0000 Waiting for PHY auto negotiation to complete......... TIMEOUT !
Could not initialize PHY ethernet@ff3f0000
Retrieving file: pxelinux.cfg/000
ethernet@ff3f0000 Waiting for PHY auto negotiation to complete......... TIMEOUT !
Could not initialize PHY ethernet@ff3f0000
Retrieving file: pxelinux.cfg/00
ethernet@ff3f0000 Waiting for PHY auto negotiation to complete......... TIMEOUT !
Could not initialize PHY ethernet@ff3f0000
Retrieving file: pxelinux.cfg/0
ethernet@ff3f0000 Waiting for PHY auto negotiation to complete......... TIMEOUT !
Could not initialize PHY ethernet@ff3f0000
Retrieving file: pxelinux.cfg/default-arm-meson-odroid-n2
ethernet@ff3f0000 Waiting for PHY auto negotiation to complete......... TIMEOUT !
Could not initialize PHY ethernet@ff3f0000
Retrieving file: pxelinux.cfg/default-arm-meson
ethernet@ff3f0000 Waiting for PHY auto negotiation to complete......... TIMEOUT !
Could not initialize PHY ethernet@ff3f0000
Retrieving file: pxelinux.cfg/default-arm
ethernet@ff3f0000 Waiting for PHY auto negotiation to complete......... TIMEOUT !
Could not initialize PHY ethernet@ff3f0000
Retrieving file: pxelinux.cfg/default
ethernet@ff3f0000 Waiting for PHY auto negotiation to complete......... TIMEOUT !
Could not initialize PHY ethernet@ff3f0000
Config file not found
ethernet@ff3f0000 Waiting for PHY auto negotiation to complete......... TIMEOUT !
Could not initialize PHY ethernet@ff3f0000
ethernet@ff3f0000 Waiting for PHY auto negotiation to complete......... TIMEOUT !
Could not initialize PHY ethernet@ff3f0000
=>

 

 

It seems to have issues to detect the partitions on the eMMC, but of course they are fine when mounting and fscking the eMMC from a different system.

Edited June 25, 2022 by MichaIng

[Franky66]

Actually not, I did not spend any more time in the last weeks and still have to buy a serial adapter for debugging. So sorry, I can't help.

[Szymon Gładysz]

@Franky66 did you managed to fix it ? I have the similar problem -> about 70% times when I'm rebooting a odroid I'm getting a emmc problem, and it seems that uboot doens't see the partitions.  But sometimes it works without problems..

[Franky66]

No sorry, switched to x86 pc and did not spend any time there.