Can't not boot after abnormal power off

[yeseul]

Hello, we contact you with Odroid booting issues.
 

I've confirmed that the problem occurs from the next boot when the board is powered off while the armbian logo is displayed. (It happens with a high probability.)
When booting, the following log is displayed and it does not boot.

 

Loading, please wait...
starting version 237
Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
Begin: Running /scripts/local-premount ... Scanning for Btrfs filesystems
done.
Begin: Waiting for root file system ... Begin: Running /scripts/local-block … done
done.
Gave up waiting for root device. Common problems:
-Boot args (cat /proc/cmdline)
  -Check rootdelay= (did the system wait long enough?)
-Missing modules (cat /proc/modules; ls /dev)
ALERT! /dev/mmcblk1p1 does not exits. Dropping to a shell!

BusyBox v1.27.2 (Ubuntu 1:1.27.2-2ubuntu3.2) built-in shell (ash)
Enter 'help' for a list of built-in commands.

(initramfs)_

After that, if you try to reboot about 5-6 times, one boot is possible. (This phenomenon does not disappear and continues.)

The sd card is not physically damaged.
This is because cloning the OS will reboot normally.

 

Another feature is that the Armbian logo does not appear normally after booting problems.
When there were no problems booting, the Armbian logo was always displayed.

It appears that some file related to booting are crashed on an abnormal shutdown.
This is because we found a file that is believed to be the cause.

 

After a few tries I successfully booted from the sd card with this boot problem and checked the /boot/armbianEnv.txt file.

And found that the file contents are broken. (The contents are different from normal files.)
I got the armbianEnv.txt file from a general sd card that did not have the problem raised in this article, overwritten it, and then booted it, and it was confirmed that it boots normally without any problem.
I think this happens if the file writes don't complete properly while boot.cmd is running. Is my guess correct?

I've heard that armbianEnv.txt is giving incorrect kernel parameters for the root filesystem root=/dev/mmcblk1p1 if it is corrupted.

On the other hand, the kernel recognizes the SD card as /dev/mmcblk0p1, so there is a problem when booting.

 

Has this phenomenon ever been reported?
I would like to know the exact cause of this phenomenon and how to fix it or prevent this problem?

We are looking for a way to recover if this happens during boot up.

For example running script that pass hard coded boot parameter to kernel each boot...

Because the product must be used in an environment where it is difficult to restrict users, unexpected power off (whether booting or not) may occur.

(However, the system must be restored and operational.)

 

We are considering mass production, so I think this is a very important issue.
Thanks in advance for the reply.
-------------------------------------------------
Environment of use
- board : odroid c4
- power : DC 9V, 0.5A
- os : ubuntu 18.04
- sdcard : SanDisk Ultra micro 32GB

 

 

Board: Not on the list

[Igor]

19 minutes ago, yeseul said:

os : ubuntu 18.04

We don't provide nor support Ubuntu 18.04 ... Armbian variants from https://www.armbian.com/odroid-c4/ (if this is some older Armbian build, make sure to update boot loader via armbian-config before proceeding. But its also possible that you are using image with stock boot loader which we don't support)

 

Latest versions, which we have in the automated testing facility shows no troubles with reboot - check here https://github.com/armbian/scripts/actions/workflows/smoke-tests.yml But Amlogic devices are picky on SD cards. Try different brand.

 

19 minutes ago, yeseul said:

We are considering mass production, so I think this is a very important issue.

 

https://forum.armbian.com/subscriptions/

[Igor]

23 minutes ago, yeseul said:

Because the product must be used in an environment where it is difficult to restrict users, unexpected power off

For this you need to rework boot process. Current OS is not ready for such use out of the box. 

[slslsl]

I'm really sorry.....

My previous post was wrong, but I can't edit it, so I'm writing again. I am using armbian OS.

 

 Hello, we contact you with Odroid booting issues.
 

I've confirmed that the problem occurs from the next boot when the board is powered off while the armbian logo is displayed. (It happens with a high probability.)
When booting, the following log is displayed and it does not boot.

 

Loading, please wait...
starting version 237
Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
Begin: Running /scripts/local-premount ... Scanning for Btrfs filesystems
done.
Begin: Waiting for root file system ... Begin: Running /scripts/local-block … done
done.
Gave up waiting for root device. Common problems:
-Boot args (cat /proc/cmdline)
  -Check rootdelay= (did the system wait long enough?)
-Missing modules (cat /proc/modules; ls /dev)
ALERT! /dev/mmcblk1p1 does not exits. Dropping to a shell!

BusyBox v1.27.2 (Ubuntu 1:1.27.2-2ubuntu3.2) built-in shell (ash)
Enter 'help' for a list of built-in commands.

(initramfs)_

After that, if you try to reboot about 5-6 times, one boot is possible. (This phenomenon does not disappear and continues.)

The sd card is not physically damaged.
This is because cloning the OS will reboot normally.

 

Another feature is that the Armbian logo does not appear normally after booting problems.
When there were no problems booting, the Armbian logo was always displayed.

It appears that some file related to booting are crashed on an abnormal shutdown.
This is because we found a file that is believed to be the cause.

 

After a few tries I successfully booted from the sd card with this boot problem and checked the /boot/armbianEnv.txt file.

And found that the file contents are broken. (The contents are different from normal files.)
I got the armbianEnv.txt file from a general sd card that did not have the problem raised in this article, overwritten it, and then booted it, and it was confirmed that it boots normally without any problem.
I think this happens if the file writes don't complete properly while boot.cmd is running. Is my guess correct?

I've heard that armbianEnv.txt is giving incorrect kernel parameters for the root filesystem root=/dev/mmcblk1p1 if it is corrupted.

On the other hand, the kernel recognizes the SD card as /dev/mmcblk0p1, so there is a problem when booting.

 

Has this phenomenon ever been reported?
I would like to know the exact cause of this phenomenon and how to fix it or prevent this problem?

We are looking for a way to recover if this happens during boot up.

For example running script that pass hard coded boot parameter to kernel each boot...

Because the product must be used in an environment where it is difficult to restrict users, unexpected power off (whether booting or not) may occur.

(However, the system must be restored and operational.)

 

We are considering mass production, so I think this is a very important issue.
Thanks in advance for the reply.
-------------------------------------------------
Environment of use
- board : odroid c4
- power : DC 9V, 0.5A
- os : Armbian 

 //uname -a

 (Linux (none) 5.10.81-meson64 #21.08.6 SMP PREEMPT Mon Nov 22 11:21:41 UTC 2021 aarch64 GNU/Linux)
- sdcard : SanDisk Ultra micro 32GB

 

 

[Igor]

8 minutes ago, slslsl said:

My previous post was wrong, but I can't edit it,

Wait two days or buy a subscription. We don't want that forum become technical support board or that too many questions are asked since it is very expensive to answer them. My time is at least equally expensive as yours.

 

For amateur usage, things works good enough. If you want that they work better, we can make them better, but read this first.

[slslsl]

Mr. Igor, I'm really sorry,

I was in a rush, so I didn't read the rules carefully.

Sorry for being rude.

Your time will of course be equal to or more expensive than mine.

I respect and appreciate the time you give.

 

I would like to know how to recover when I have this boot problem. Even if the problem occurs because the user accidentally cuts the power, we want to recover it on the next boot.

Reading your answers, it seems that modifying the boot-related scripts solves the problem. Is that correct?
(ex) Edit /boot/boot.cmd... etc.)

 

 

I apologize once again for any disruption to the forum operation.

[CryBaby]

This sounds like a common problem seen with Raspberry Pi. There it is due to the boot partition being formatted as VFAT. If that is also the case with the Odroid then the solution is to modify /etc/fstab to mount /boot read only.

It can also help to add fsck.mode=force to the kernel command line so the fs gets checked on every boot.

[lampra]

11 hours ago, CryBaby said:

fsck.mode=force

This will only check, but it will not repair.

As they use Ubuntu, they should probably consider `overlayroot`

To my knowledge, by default, armbian uses `commit=600` in fstab to minimize corruption probability

[slslsl]

On 2/4/2022 at 10:34 PM, CryBaby said:

This sounds like a common problem seen with Raspberry Pi. There it is due to the boot partition being formatted as VFAT. If that is also the case with the Odroid then the solution is to modify /etc/fstab to mount /boot read only.

It can also help to add fsck.mode=force to the kernel command line so the fs gets checked on every boot.

 

Thank you for answer. I have an additional question for you.

1. Is the problem caused by VFAT? I wonder what the reason is.

2. If the /boot directory is mounted as read only, is it correct that all files in the /boot directory become unmodifiable?

[slslsl]

On 2/5/2022 at 9:56 AM, lampra said:

This will only check, but it will not repair.

As they use Ubuntu, they should probably consider `overlayroot`

To my knowledge, by default, armbian uses `commit=600` in fstab to minimize corruption probability

 

Thankyou for the answer!

Can Armbian also use overlay boot? Is the result you want to achieve with overlayroot the same as mounting /boot read-only?

Also, is it true that commit 600 means data writes occur every 6 seconds, which is enough time to protect the boot system?

[CryBaby]

47 minutes ago, slslsl said:

Is the problem caused by VFAT? I wonder what the reason is.

I don't know about Odroid, on the Raspberry Pi is certainly is a problem. (V)FAT is a very bad filesystem that was designed by idiots Microsoft in the 1980s. It is very prone to losing metadata if the power goes out. Raspberry Pi uses it as its proprietory boot loader cannot read anything else.

 

52 minutes ago, slslsl said:

If the /boot directory is mounted as read only, is it correct that all files in the /boot directory become unmodifiable?

Yes, that's why it prevents damage on loss of power. You usually only need to write to /boot when you update the kernel or bootloader. You can temporarily remount it read/write before doing an update then when you reboot it will be read-only again.

[slslsl]

44 minutes ago, CryBaby said:

I don't know about Odroid, on the Raspberry Pi is certainly is a problem. (V)FAT is a very bad filesystem that was designed by idiots Microsoft in the 1980s. It is very prone to losing metadata if the power goes out. Raspberry Pi uses it as its proprietory boot loader cannot read anything else.

 

Yes, that's why it prevents damage on loss of power. You usually only need to write to /boot when you update the kernel or bootloader. You can temporarily remount it read/write before doing an update then when you reboot it will be read-only again.

 

 

Oh I see. Thanks for your kind reply!!!!!!!!!!!

 

I took your advice and looked up my /etc/fstab.

My root directory is already mounted as read-only as shown below

(The file system is not VFAT, but ext4)

 

UUID=XXX-XXX-... / ext4  defaults,noatime,nodiratime,commit=600,errors=remount-ro 0 1
tmpfs /tmp tmpfs defaults,nosuid 0 0

 

 

 

Doesn't above apply to the subdirectory /boot? If it is not applicable, can I add the following command to /etc/fstab?

UUID=XXX-XXX-... /boot             vfat errors=remount-ro 0       1

 

And I don't really know what the behavior might be different after mounting it like this.... Can you answer this?

 

Thankyou!

 

[CryBaby]

If there is not already a line for /boot in your fstab then it is a subdirectory not a separate partition. In which case the read-only trick is not applicable. Just adding a line for it will not work.

 

I would try adding fsck.mode=force fsck.type=repair to your kernel command line. In most cases that will be able to fix any corruption on boot.

 

If you still have the problem overlayroot is the next thing to try. As I understand it that will make your real root filesystem read-only and any changes are only made to a copy of it in RAM. So a power loss will lose those changes but you are guaranteed a working root to boot off. Never set it up myself so Google it.

[CryBaby]

15 hours ago, slslsl said:

errors=remount-ro

This does not mean it is normally mounted read-only. It means it will be mounted read-only if mounting it read-write fails.

[lampra]

19 hours ago, slslsl said:

Armbian also use overlay boot?

Yes Armbian, Ubuntu flavor can use overlayroot.

 

19 hours ago, slslsl said:

commit 600 means data writes occur every 6 seconds

https://unix.stackexchange.com/questions/155784/advantages-disadvantages-of-increasing-commit-in-fstab

 

[slslsl]

7 hours ago, CryBaby said:

If there is not already a line for /boot in your fstab then it is a subdirectory not a separate partition. In which case the read-only trick is not applicable. Just adding a line for it will not work.

 

 

Yes, that's right! 

Since the root directory is already mounted, I did not remount /boot.

Instead, I changed the permission of /boot to read-only, or tried to change the permission of the armbianEnv.txt file to read-only.

However, this method did not solve the boot failure.

If that cannot write to armbianEnv.txt, that may not be able to properly pass boot parameters to the kernel. (When I search my fstab with cat in recovery mode, nothing comes up.)

 

8 hours ago, CryBaby said:

would try adding fsck.mode=force fsck.type=repair to your kernel command line. In most cases that will be able to fix any corruption on boot.

I will try it! Thankyou!

 

8 hours ago, CryBaby said:

If you still have the problem overlayroot is the next thing to try. As I understand it that will make your real root filesystem read-only and any changes are only made to a copy of it in RAM. So a power loss will lose those changes but you are guaranteed a working root to boot off. Never set it up myself so Google it.

Great. If the previous method fails, I will try the overlayroot method.

 

Actually I just fixed boot.cmd now and it solved the boot fail problem.

Make a backup file such as armbianEnvDefult.txt that is not accessed during booting, and load it in boot.cmd.

In this case, even if the original armbianEnv.txt is damaged, booting does not fail. But I don't think this method is very cool... isn't it?

 

Than I will also try to apply your advice!

I really appreciate the advice!!

[slslsl]

6 hours ago, lampra said:

Yes Armbian, Ubuntu flavor can use overlayroot.

 

https://unix.stackexchange.com/questions/155784/advantages-disadvantages-of-increasing-commit-in-fstab

 

 

 

Thanks for the reply, I'll consider overlayroot!

[sanchez911]

slslsl

 

Same problem, but we're using Asus Tinkerboard. Although error is different, "ALERT! UUID=..... does not exist". I had not time to investigate it yet, but maybe it's something about swap partition. Damned problem. We're also using these boards in industrial equipment and this fail causes big troubles for us.

 

P.S. I just wonder how this problem is used on real industrial PCs, or is there any raspberry form-factor industrial board..

[CryBaby]

I prefer to use LABEL= in my fstab rather than UUID. It is a little more forgiving of media changes, provided you remember to give your filesystems consistent labels. You probably don't want to have swap on an SD card. I thought Armbian used zram for that.