Problems rebooting amlogic based boxes from emmc


Recommended Posts

This issue is specifically about an issue where 'sometimes' a box will fail to boot off of emmc (after having booted successfully previously).  The workaround is to boot once from SD card and then the emmc boot generally works fine again.

 

I am starting this thread as a central place to discuss this issue that has been reported by a few people that I know. @lgranie and @robertoenr both have reported this problem either recently or in the past and I see it myself frequently.

 

It would appear that something is getting set in the uboot environment that is causing the boots to fail and fixed/cleared by a boot from SD card.

I haven't had time to look at this and don't know when/if I will have time.  But I wanted to log this so others would know about the issue and perhaps someone might have time to dig into this.

Link to post
Share on other sites
Donate and support the project!

Thank you @SteeMan for creating a new dedicated thread with this issue.

 

I give the most detailed information in my hand, and tests in order to keep track in context:

 

1. Details about the TV Box:

  • Model: TX3 Mini (new model 2021).  It's the genuine.
  • Board: 1.6
  • Processor: Amlogic S905W
  • GPU: Mali 450
  • WIFI module: AP6330

Links for more TV Box information (in vietnamite):

https://tinhte.vn/thread/tong-hop-cac-ban-firmware-cua-android-tv-box-tx3-mini.3022766/

https://tinhte.vn/thread/mo-hop-tx3-mini-2021-phien-ban-phan-cung-moi.3216425

https://tinhte.vn/thread/firmware-android-tx3-mini-phien-ban-2021.3229154/

 

Some board photos for reference.

https://mega.nz/file/SBgATZaa#iZOMpvxc0g_NfiOjxfnJjotEqDNnrM2vapUmNm_08Cc

https://mega.nz/file/TNhUEJzD#Tj_VAakZrYM0L9itPjCFwV7MjVyplIXcehMZkVmrOYM

https://mega.nz/file/SF5WxRxa#clOe168ZugDJOQkNRmxpSdW9HlrT7v6vM6aZiZTpiHI

https://mega.nz/file/nE4glBTB#yDxjIpCQ5Te_GyiHp9fzQGxI1Wn_JnQjo2pGnpokK_w

 

Note: The memory chip confirmed is eMMC if you find the model.

 

2. Details about the Armbian image and installation

I used this image file "Armbian_20.10_Arm-64_focal_current_5.9.0_desktop.img.xz", downloaded from your mega repository. And used BalenaEtcher to burn it to a fast USB drive (SanDisk Ultra Fit 64GB model USB 3.0).

After burn the image on the USB drive, these are the steps I follow:

 

2.A) Edit the file "/extlinux/extlinux.conf" with these contents:

LABEL Armbian
LINUX /zImage
INITRD /uInitrd

# aml s9xxx
FDT /dtb/amlogic/meson-gxl-s905w-tx3-mini.dtb
APPEND root=LABEL=ROOTFS rootflags=data=writeback rw console=ttyAML0,115200n8 console=tty0 no_console_suspend consoleblank=0 fsck.fix=yes fsck.repair=yes net.ifnames=0

 

2.B) Duplicate the file "u-boot-s905x-s912", and rename to "u-boot.ext".

 

2.C) Insert the USB drive on the TV Box and power it on, while pressing the reset button with a pin until Armbian boots (this is my method to enable multi-boot).

 

3.C) Once started armbian, in order to make the wifi module works:

sudo mv /lib/firmware/brcm/brcmfmac4330-sdio.txt /lib/firmware/brcm/brcmfmac4330-sdio.txt.old
sudo ln -s  /lib/firmware/brcm/brcmfmac-ap6330-sdio.txt /lib/firmware/brcm/brcmfmac4330-sdio.txt

 

3.D) Finally I installed to eMMC, executing the /root/install-aml.sh script. Once finished with success, powered off the device, remove the USB drive, and start the TV Box with nothing connected. It seems everything works OK!

 

3. Issue with linux power off commands.

After using armbian on the TX3 mini during a time, I noticed that the commands like "sudo poweroff" and "sudo shutdown -h now" doesn't work. Instead it's restarting. So it's impossible to power off the device. The only way is to plug off directly from wall. That's what makes travel me to the next issue.

I don't know if we can solve it in someway...

 

4. Issue on boot (randomly).

After explaining the previous issue, I keep testing by switching the TV Box from wall and power on once a day, because it's the only way to power it off.

I noticed that sometimes (randomly, not always), when I plug off from wall power the device, the next time when I power it on doesn't boot, showing only a black screen (not showing the TX3 Mini Splash and not showing the "hit any key to stop autoboot..." message). 

When this issue happen, I perform the following steps to solve:

1. Power off the device and plug a configured USB drive with Armbian. This is the same USB drive from where I installed initially Armbian.

2. Power on the device (without pressing any button) and the system starts OK from eMMC.

3. Power off the device and unplug the USB drive.

4. Power on the device without nothing connected. It boots OK from eMMC.

 

As it said @SteeMan it seems that the uboot is setting something randomly, and when you start with the media, it gets recovered/writed again.

I believe that it's not memory corruption or memory lost, because it would crash entire sectors and the memory would be a mess. This is not the case, once recovered, the system works back perfect without no issue.

 

Note: also tried with the "meson-gxl-s905w-p281.dtb" and the issues are the same.

 

If anyone can help to trace this issues, would be nice, really appreciated.

 

Thanks!

Link to post
Share on other sites

Update:

 

Following the following topic from @erew, I've check his section "LOST.DIR, System Volume Information and Android folders cause problems booting off eMMC/NAND.":

https://forum.armbian.com/topic/10685-installation-guide-and-your-questions-answered-for-tv-boxes/

 

 

So, I've check that on my /boot partition on eMMC, with "ls -a", there was a folder called "LOST.DIR". I have deleted it and reboot.

 

@SteeMan and @lgranie if you want to also test, check that you don't have any of this hidden folders on the boot partition on eMMC. I have no hope about solving with this solution, but you have to start something...

Anyone has UART access on the TV Box to check the log during u-boot and previous process, when the problem happens? I have tried but my TX3 Mini model has a really tiny pads for RX/TX/GND and I have broken them trying to solder the wires... :unsure:

 

Keep testing if the random reboot failure continues...

Link to post
Share on other sites

Update 25/01/2021:

The attempt described on previous post doesn't work. I have a new boot fail today.

 

Method to reproduce the boot fail

My method to reproduce the boot fail is to leave the TV Box on during almost 48h. Also I have a service that writes rows each second to a PostgreSQL database. After 48h of continuous work, I plug off the TV Box from wall. Then keep the TV Box off during almost 3h. When you plug on again... BOOM here is the fail.

 

Fail cause suspect

Taking from base that my TV Box is writing to eMMC during 48h of work and I plug off from the wall without switching off correctly. I suspect that the boot problem happens when the system has too many data writed to disk or probably disk pending transactions. It may cause the /boot partition on eMMC gets trashed?

 

Test to identify key files

Because inserting a pre-configured external media (in my case only from USB) and boot with it makes the TV Box to life again, I've tried to identify what files on external media makes the device boot again from eMMC.

 

My test consists in start with external media /boot partition emtpy, try to boot, add a file, try to boot, and so on:

 

  • Insert the external media with /boot partition empty, and power on the TV Box. Result: NOT BOOTING
  • Add the following folders to external media /boot partition: “dtb” “extlinux” y “LOST.DIR”. And power on the TV Box. Result: NOT BOOTING. 
  • Add the following files to external media /boot partition: “u-boot-s905”, “u-boot-s905x-s912”, “u-boot-s905x2-s922”, “u-boot.ext”, “uInitrd”, “zImage”. And power on the TV Box. Result: NOT BOOTING.
  • Add the following files to external media /boot partition: “System.map-5.9.0-arm-64”  And power on the TV Box. Result: NOT BOOTING.
  • Add the following files to external media /boot partition: “boot.bmp” y “s905_autoscript”   And power on the TV Box. Result: NOT BOOTING.
  • Add the following files to external media /boot partition: “initrd.img-5.9.0-arm-64”. And power on the TV Box. Result: NOT BOOTING.
  • Add the following files to external media /boot partition: “config-5.9.0-arm-64”. And power on the TV Box. Result: NOT BOOTING.
  • Add the following files to external media /boot partition: “armbian_first_run.txt.template” y “boot.ini”. And power on the TV Box. Result: NOT BOOTING.
  • Add the following files to external media /boot partition: "aml_autoscript” y “aml_autoscript.zip” . And power on the TV Box. Result: YEAH, BOOTS FROM EMMC!!

Note that the file "emmc_autoscript" is the unique that hasn't been copied to /boot partition of external media on the tests, so I suppose it is not necessary to re-life the TV Box.

 

Summary: the "aml_autoscript” y “aml_autoscript.zip” are the first searched by the TV Box when this problem happens. I suppose that the behind code of this files makes the TV Box starts... 

This is the code on the "aml_autoscript":

Why the TV Box gets recovered after that and starts again from eMMC without the external media, it's still a mystery...

 

If anyone who has more advanced knowledge about this boot system, could give some light...?

Link to post
Share on other sites

More tests! Update 07/02/2021:

 

TV Box is the same as described on https://forum.armbian.com/topic/16837-problems-rebooting-amlogic-based-boxes-from-emmc/?do=findComment&comment=117801

 

Some tests with recent and historical versions of Armbian from @balbes150

 

TEST 1. Try Armbian 20.02

  1. Install Stock firmware on the TV Box with Amlogic USB Tool. After that, boot Android. Finally plug off the AC plug.
  2. Boot "Armbian_20.02.0-rc1.037_Aml-s9xxx_bionic_current_5.5.0-rc6_desktop_20200205.img" from USB drive using DTB "meson-gxl-s905w-tx3-mini.dtb". included on the image. It boots OK.
  3. Install on eMMC with the script on /root/install-aml.sh. Finish OK.
  4. Power off device with "sudo poweroff".  When the shutdown process finish, eject the USB Drive. It try to reboots instead of powering off.

Note: with Armbian 20.02 and it's dtb for TX3 Mini it only detects 1GB of RAM (only 965MB). The same with p281 dtb.

Result: the TV Box doesn't boot from eMMC, it keeps with black screen forever (it has HDMI signal).

 

TEST 2. Try Armbian 20.05.06

  1. Install Stock firmware on the TV Box with Amlogic USB Tool. After that, boot Android. Finally plug off the AC plug.
  2. Boot "Armbian_20.05.6_Arm-64_focal_current_5.7.2_desktop_20200612.img.xz" from USB drive using DTB "meson-gxl-s905w-tx3-mini.dtb". included on the image. It boots OK.
  3. Install on eMMC with the script on /root/install-aml.sh. Finish OK.
  4. Power off device with "sudo poweroff".  When the shutdown process finish, eject the USB Drive. It try to reboots instead of powering off.

Note: with Armbian 20.05.06 and it's dtb for TX3 Mini it only detects 1GB of RAM (only 965MB). The same with p281 dtb.

Result: the TV Box doesn't boot from eMMC, it keeps flickering the screen, like a loop.

 

TEST 3. Tests on Armbian 20.10:

  1. Install Stock firmware on the TV Box with Amlogic USB Tool. After that, boot Android. Finally plug off the AC plug.
  2. Boot "Armbian_20.10_Arm-64_focal_current_5.9.0_desktop.img.xz" from USB drive using DTB "meson-gxl-s905w-tx3-mini.dtb". included on the image. It boots OK.
  3. Install on eMMC with the script on /root/install-aml.sh. Finish OK.
  4. Power off device with "sudo poweroff".  When the shutdown process finish, eject the USB Drive. It reboots instead of powering off.
  5. Boots OK from eMMC. Once started I perform several test of restart/reboot/shutdown with keyboard and mouse connected:
    1. Restarted 3 times OK from Graphical Interface. When it boots, shows the TX3 Mini boot logo.
    2. Powered off 3 times OK from Graphical Interface. It restart instead. When it boots, shows the TX3 Mini boot logo.
    3. Powered off several times by disconnecting the plug on different momments (while U-boot starts, while booting, while loading desktop...). When plug on it boots OK and shows the TX3 Mini boot logo.
    4. Restarted 5 times from terminal with "sudo reboot". When it boots, shows the TX3 Mini boot logo.
    5. Powered off 5 times from terminal with "sudo shutdown -h now". It restart instead. When it boots, shows the TX3 Mini boot logo.
    • NOTE: In all previous reboots/shutdowns it shows the following message during 1min 32 sec: "a stop job is running for ifup for eth0". So the shudown process was slow.
  6. Execute the following lines in order to make the wifi AP6330 works, but still I not connect to any WIFI network: "sudo mv /lib/firmware/brcm/brcmfmac4330-sdio.txt /lib/firmware/brcm/brcmfmac4330-sdio.txt.old and sudo ln -s /lib/firmware/brcm/brcmfmac-ap6330-sdio.txt /lib/firmware/brcm/brcmfmac4330-sdio.txt".
    1. Execute "sudo reboot". It reboot OK. When boot it shows the TX3 Mini boot logo.
    2. Reboot from Graphical Interface. It restart instead. When it boots, shows the TX3 Mini boot logo.
    3. Powered off 3 times OK from Graphical Interface. It restart instead. When it boots, shows the TX3 Mini boot logo.
    4. Powered off several times by disconnecting the plug on different momments (while U-boot starts, while booting, while loading desktop...). When plug on it boots OK and shows the TX3 Mini boot logo.
    • NOTE: In all previous reboots/shutdowns it shows the following message during 1min 32 sec: "a stop job is running for ifup for eth0". So the shudown process was slow.
  7. Connect to my WIFI network in order to the TV Box have Internet connection:
    1. Power off rom Graphical Interface. It try to restart instead. When it boots it keeps forever on black screen (with HDMI signal) and doesn't boot again. 
    2. In order to make the TV Box boots again, I have to boot it with the USB Drive with Armbian 20.10 installation, and after power off, eject USB drive. After this it boots OK again, but doesn't show anymore the TX3 Mini Boot logo. The crash make dissappear the boot logo.
    • NOTE: In the last power off before the black screen, is the first time that the TV Box doesn't show the message "a stop job is running for ifup for eth0", so the power off was fast.

Note: with Armbian 20.10 and it's dtb for TX3 Mini it detect full RAM (2GB), exactly 1,82GB.

Result: it doesn't boot again, after connect it to internet. When crash the boot logo disappeared and the "a stop job is running for ifup for eth0" also disappeared.

 

 

TEST 4. Repeat tests on Armbian 20.10:

  1. Install Stock firmware on the TV Box with Amlogic USB Tool. After that, boot Android. Finally plug off the AC plug.
  2. Boot "Armbian_20.10_Arm-64_focal_current_5.9.0_desktop.img.xz" from USB drive using DTB "meson-gxl-s905w-tx3-mini.dtb". included on the image. It boots OK.
  3. Install on eMMC with the script on /root/install-aml.sh. Finish OK.
  4. Power off device with "sudo poweroff".  When the shutdown process finish, eject the USB Drive. It reboots instead of powering off.
  5. Boots OK from eMMC. Once started I perform several test of restart/reboot/shutdown with keyboard and mouse connected:
    1. First time, I reboot from Graphical Interface. It doesn't reboots. It lost HDMI signal and doesn't boots, but the TV Box keeps on. I must power off by unplugging the power plug. When power on again it boots OK, and shows the TX3 Mini boot logo.
    2. Second time, I reboot again from Graphical Interface. Same fail as previous.
    3. Third time, I reboot again from Graphical Interface. It reboots OK, and shows the TX3 Mini logo on boot.
    4. Fourth time, I reboot again from Graphical Interface. Same fail as previous.
    5. Fifth time, I reboot from terminal with command "sudo reboot".  It reboots OK, and shows the TX3 Mini logo on boot.
    6. Sixth time, I reboot again from terminal with command "sudo reboot".  It lost HDMI signal and doesn't boots, but the TV Box keeps on. I must power off by unplugging the power plug. When power on again it boots OK, and shows the TX3 Mini boot logo.

Note: with Armbian 20.10 and it's dtb for TX3 Mini it detect full RAM (2GB), exactly 1,82GB.

Result: the boot system is irregular under the same conditions. Sometimes it lost HDMI signal; other times it keeps with black screen.

 

It could be caused by buggy u-boot? Anyone has the base source code of u-boot from balbes150 in order to try to recompile?

 

I found this post, but I don't know if could be compatible with armbian:

https://www.codedbearder.com/posts/mainline-linux-on-tx3-mini/

Edited by robertoenr
Link to post
Share on other sites

Partial solution to the problem!

I've upgraded the stock firmware+bootloader to another more recent (Android 9 with kernel 3.14 oficial) with Amlogic USB Flash tool, checking the option "Erase bootloader". After this, I've booted completely Android.

 

Next , I've booted Armbian, installed the WIFI, change the keyboard layout, timezone and hostname. And installed on eMMC.

 

Now, with Armbian 20.10 Focal 5.9 installed on eMMC, I've restarted hundreds of times and the "black-screen-non-booting" problem has disappeared.

 

So, if you don't have more official firmwares for your TV Box, try any unofficial ROM like ATV Experience. It seems that changing the bootloader and binary blobs can solve it.

 

My issue now is that sometimes on reboot, the screen doesn't get HDMI signal from TV Box and it doesn't boot. It boots OK if I unplug and plug again the power plug.

 

Anyone knows about this strange HDMI restarting issue?

 

Link to post
Share on other sites

Hi, I have a complete solution to the problem, at least in my case with TX3 Mini 2021 with s905w cpu, 1.6 board, and AP6330 wifi chip. Summary tips:

  • Install an Android stock firmware with kernel 3.XX. It seems that usually on Android versions with this kernel the bootloader is older and looks "more compatible" with Armbian.
  • Use Armbian 20.05.6 with kernel 5.7.2, and modify the install script, locate this part and update to this:
parted -s "${DEV_EMMC}" mklabel msdos
parted -s "${DEV_EMMC}" mkpart primary fat32 1000M 1512M
parted -s "${DEV_EMMC}" mkpart primary ext4 1513M 100%

 

  • Freeze Armbian kernel updates with armbian-config (note that if you upgrade kernel to 5.9.0 or 5.9.9 the "HDMI-non-signal-reboot" issue appears again).

 

On my tests during 2-3 weeks, with auto scripts to reboot periodically the TV Box, I have not seen the reboot issue again. So I consider solved in my case.

Link to post
Share on other sites
On 1/23/2021 at 12:34 PM, robertoenr said:

Update:

 

Following the following topic from @erew, I've check his section "LOST.DIR, System Volume Information and Android folders cause problems booting off eMMC/NAND.":

https://forum.armbian.com/topic/10685-installation-guide-and-your-questions-answered-for-tv-boxes/

 

 

So, I've check that on my /boot partition on eMMC, with "ls -a", there was a folder called "LOST.DIR". I have deleted it and reboot.

 

@SteeMan and @lgranie if you want to also test, check that you don't have any of this hidden folders on the boot partition on eMMC. I have no hope about solving with this solution, but you have to start something...

Anyone has UART access on the TV Box to check the log during u-boot and previous process, when the problem happens? I have tried but my TX3 Mini model has a really tiny pads for RX/TX/GND and I have broken them trying to solder the wires... :unsure:

 

Keep testing if the random reboot failure continues...

 

Removing the 'System Volume Information' directory in /boot seems to have fixed the reboot issue for me. Will continue to test over time but I'm cautiously optimistic. Thanks for the pointer!

Link to post
Share on other sites
On 4/13/2021 at 12:41 PM, mcb777 said:

 

Removing the 'System Volume Information' directory in /boot seems to have fixed the reboot issue for me. Will continue to test over time but I'm cautiously optimistic. Thanks for the pointer!

Back to report that this does make a difference but there are still times it won't reboot properly in which case I need to boot it from a USB stick once and then it'll boot again from eMMC

Link to post
Share on other sites