Separate boot partition and different root partitions → simple man’s A+B update possible?

[LanMarc77]

Getting into buildroot and yocto seems a bit too much for me to get an A+B style update possibility. I would like to stick with the quality images provided by Armbian.

The following applies to a NanoPi-NEO3 others SBC might have different partition layouts.

 

From what I understood is that uboot looks for /boot in /dev/mmcblk0p1 reads the boot.cmd/scr and armbianEnv.txt and then boots the kernel accordingly.

The armbianEnv.txt seems to point the kernel to the root partition using the rootdev directive.

 

If /dev/mmcblk0p1 would now only hold /boot would it be possible to point to a different partition for the rootfs using armbianEnv.txt?

If so I could imagine changing the partition layout to the following:

 

/dev/mmcblk0:

/dev/mmcblk0p1: boot partition

/dev/mmcblk0p2: rootfs partitionA

/dev/mmcblk0p3: rootfs partitionB

/dev/mmcblk0pX: multiple more partitions that survive updates

 

Too now allow consistency with the rootfs partitions also the boot partition directory structure would slightly change. It would contain 2 directories: /bootA, /bootB AND a hardlink /boot which either points to /bootA or /bootB

In the running rootfs the fitting /boot is mounted via a bind mount hoping normal apt-get updates can deal with this. This would allow creating a rootfs by updating and verifying everything works locally and then creating an image from it.

 

Now to OTA a relatively simple script running from rootfsA can download a new rootfs image/file structure and place it in the partition of rootfsB. Same with the new boot partition just into /bootB. The values of the armbianEnv.txt from the new boot image always point to the corresponding rootfs partition either because the image was already built for this or the script dynamically adjusts them. Same goes for fstab, machine-id, ssh identity and other rootfs specifics. The script will then verify that the write worked to avoid sd card issues.

The (almost) atomic operation for switching the systems would be changing the hardlink of /boot to the /bootB directory and reboot.

 

This idea will not detect any boot issues and revert automatically back as uboot is not involved. Also this only works if uboot is still compatible with the new boot files provided. But an additional new uboot image could also be provided during the update.

Reverting can be done manually (relatively fast) by changing the hardlink back and rewriting a previously created backup copy of a might be updated uboot image.

 

Independent of the fact that buildroot, yocto, rauc, mender… systems have a better feature set, do you think this could work?

[eselarm]

19 minutes ago, LanMarc77 said:

If /dev/mmcblk0p1 would now only hold /boot would it be possible to point to a different partition for the rootfs using armbianEnv.txt?

That is basically what I have been doing. I reconstructed image/partitions to have bootfs and rootfs separated. Typically format bootfs as simple FAT where you can fix things even on Windows computers maybe when SD-cards. For new fast SBCs I just have many partitions on NVMe where various Linux distros rootfs are and just copy or rename armbianEnv.txt which is then essentially only a UUID of a rootfs.

 

When I know the SBC well enough, it is more the kernel only that matters, so i made a extlinux.conf generator so I can select a kernel via U-Boot serial console.

 

I don't see what A+B brings me as I use Btrfs for rootfs and use snapper to make 'last-known-good' snapshots and also transfer those to NAS or so. If you want totally unattended, then A+B is an option, look at Android as well I would say.

But more towards PC like systems, one can put efi bootloader (efi-grub) on the bootFAT, that allows you to select last-known-good more or less if you deploy snapshots. I usually do update in-place, so more like rolling release, but you can also do updates on a new read-write snapshot and fallback if it would fail to boot, see https://kubic.opensuse.org/documentation/man-pages/transactional-update.8.html  for example how that works.

 

U-Boot also has options I think to get to last-known-good automatically, but I don't know how that would work. For my stuff at home, it is good enough that I can quickly go back to older snapshot and that works for more than a decade on my PCs, so I do the same on SBCs/embedded, as long as they have a serial console cable (or HDMI,keyboard). For the NanoPi-NEO I also had Btrfs capable U-Boot, so then U-Boot can directly boot a certain rootfs partition, but it is not really default, so I keep an extra bootFAT.

 

[LanMarc77]

So am I reading correctly from your post that separating boot and rootfs using armbianEnv.txt does work?

Do you think the hardlink option inside the boot partition could work?

 

I would most likely test this setup but would like to know before if there are issues why it never would work to avoid work,

I know there are many other options including an initramfs with btrfs support to avoid partitions for different rootfs to then use subvolumes. Still I would like to stick for the time being with the original idea.

Thanks.

[eselarm]

13 hours ago, LanMarc77 said:

So am I reading correctly from your post that separating boot and rootfs using armbianEnv.txt does work?

Do you think the hardlink option inside the boot partition could work?

armbianEnv.txt sets parameters for the U-Boot bootloader. In theory you can also manually type it all via serial console at U-Boot prompt. Separating boot and rootfs is independent of that.

 

If you use Ext4 for bootfs, you can make hardlinks (or symlinks). That is also what is done by kernel install script if /boot is on Ext4, else copies are done (when FAT).

Edited August 25 by eselarm

[LanMarc77]

Thanks @eselarm for the input that this should work. So I tested it. And it does!

I leave a few hints for anyone who wants to replicate.

When creating a new image layout working directly with the image files was the fastest for me. So you need to download one original Armbian image. I used a minimal cli image.

 

Image preparations

Create a new image file (e.g. with dd) that will be partitioned like described above.

Copy all the first sectors until the first partition starts from the original image to the new image. This will copy the original partition table but more importantly also the uboot bootloader into the new image (32768 sectors = 16MB).

 

Partitioning

Partition the new image as described above (e.g. with fdisk/sfdisk) and format the new partitions. One smaller boot partition (e.g. 512MB, ext4) and two bigger root partitions (e.g. 3GB, ext4). I will use the terms rootA and rootB for the two new root partitions in the new image from here on. Get the UUIDs of the newly created boot and rootA partition of the new image (e.g. lsblk/blkid).

Most likely instead of using two separate ext4 rootA and rootB partitions one bigger btrfs partition and the usage of subvolumes could be possible. But I did not test this.

 

Copy boot and adjust

Mount the new boot partition and create directories bootA and bootB and a relative symlink boot pointing to bootA. Place all files from the original image’s /boot directory into bootA of the new image. Adjust the UUID line in armbianEnv.txt that still points to the original image’s root partition to the rootA UUID from the new image.

 

Copy root and adjust

Mount rootA and copy all the files from the original image into it. Remove all files from the /boot directory in rootA as we will mount the boot partition into this directory anyway and do not want to get mixed up. Create a new directory /rawBoot in rootA.

Edit the fstab of rootA and change the UUID that still points to the original image’s root partition to the rootA UUID. Add a line to fstab to mount the boot partition of the new image into /rawBoot. Add a line to fstab to bind mount the directory /boot of /rawBoot into /boot of rootA.

Add an empty file /root/.no_rootfs_resize in rootA to avoid the partitions being changed if the original image was never booted before.

 

I did all of the above also with an archived older Armbian image and put that into rootB (and bootB).

With this setup we now have two separated OS versions. They can be switched by simply changing the boot symlink to bootA or bootB and reboot. Because we used bind mount we can also do kernel upgrades which end up in the correct directory.

As all of the steps are easily automatable by a shell script we can now download a new earlier prepared update package place its content into the other boot directory and partition adjust their UUIDs (and maybe copy SSH keys and machine-id if we want to keep system identity) and change a symlink.

If we need to go back to the older image just the symlink needs to changed back. Either from within a running system or if that is not possible by taking the card out and doing this externally.

 

As I said earlier this A+B update scheme is not as robust as other tools. But we can stay with prebuilt images from Armbian that can be locally configured the way we want it. After this an update package can be created and distributed which could also contain a new uboot. If this is tested well before distribution I guess most errors that make the other tools so robust can be avoided.

 

[laibsch]

I will soon be in the situation of having an offsite Helios4 installation at my non-techie father's place on another continent.  I want to prepare for being able to do remote updates and if possible even some Armbian image testing.  For that, I need a robust, bullet-proof and fully remote mechanism.  The most I should assume to be able to ask my father is to unplug and replug the machine to powercycle.  I hope you guys don't mind me joining your conversation to discuss my options.

 

In the past, I had a setup on X86 with grub that I would like to hopefully replicate now in function with the Helios 4.  There was a very minimal (read-only?) OS in a partition (network with DHCP, sshd) that would be the default boot target.  The machine would always boot into this after a power cycle.  From there, I issued a grub-reboot command to reboot into my desired OS.  If I screwed the main system up somehow, I was always able to go back into the minimal system.  How would I go about doing that in the Armbian world?  From your discussion, it seems to be about tweaking armbianEnv.txt.  But how can I do that in a way that has a bullet-proof option the system falls back to after a power cycle but also allows me to tweak it in a way that survives a warm reboot?

I do like btrfs a lot and believe it should play a role here.  I read @eselarm mentioning btrfs support in u-boot.  How can I verify which u-boot is installed on my machine and whether it supports btrfs or not?

 

Edit:

https://embear.ch/posts/sw-update-concepts/

https://rauc.io/

https://mender.io/

https://bootlin.com/pub/conferences/2022/elce/opdenacker-implementing-A-B-system-updates-with-u-boot/opdenacker-implementing-A-B-system-updates-with-u-boot.pdf

[Igor]

We have a long wish to implement OTA solution into the build framework. The problem is - like always - time / not enough man power to run another (sub)project.

 

Instead of reinventing the wheel - better is to follow / use (learning from) experts made frameworks:
https://rauc.io/ as a base perhaps also this https://github.com/silitics/rugix/issues/82 and https://qbee.io/docs/index.html - all as extensions.

 

If we can join resources, to do it right, its easier. But its still timely and complex due to the nature of what Armbian is dealing with.

[The Tall Man]

I'd read rumors that uBoot has the ability for a multi-boot menu, but I was never able to get any video out of it to my DVI monitor to explore that too deeply.

 

What I ended up doing was installing this UEFI port, and booting from it (instead of uBoot):

https://github.com/edk2-porting/edk2-rk3588

 

The built image is about 6.6 MB, but the image actually uses 9 MB:. It consists a GPT table for a 16 MB device, plus a portion of an 8 MB partition (no flags) who's name must be uboot. The difference within the 8 MB partition is where it stores the user configuration settings. So I think the easiest and most secure way to flash the image is to first promote its size to 9 MB before flashing it to the microSD card or SPI-NOR (I've done each with good results):

 

sudo dd if=/dev/zero of=final-for-flashing.img bs=1M count=9
sudo dd if=RK3588_NOR_FLASH.img of=final-for-flashing.img bs=512 conv=notrunc

 

There are a few other versions of it for other hardware (other than rk3588).

https://github.com/orgs/edk2-porting/repositories?type=all&q=edk2

 

That will boot UEFI and into grub very easily, and I do use grub to select which OS to boot into. I have one grub partition, and multiple installed OS partitions.

 

The edk2 port is highly configurable, and recognizes hardware, including the network. Not sure if it's usable for remote access or not in its current build, but I think it can, at least, boot off the network.

 

You can also configure a standard layout/path for your devicerees and overlays, where it will load them, but I haven't tested that out yet. If you don't override in the edk2 configuration or in grub, it will use the Vendor or Edge kernel's devicetrees (built in), or ACPI (also built in). Btw, I built it last night with a tiny change I made to the Orangepi5-plus edge devicetree to re-enable the ES8388 audio, and tested it out in Trisquel 12 with its kernel (with the devicetree from edk2) - it worked! The great thing is - their git clone and build process puts ALL the source files on your computer, so you don't need the internet to build, and you can know that any changes you make in the source code won't be "updated" overwritten by the build process.

 

I've found with Grub, the update-grub utility will not find and add devicetrees, so you have to do that manually or hack in (which is what I've done). But again, apparently the edk2 port would make that unnecessary.

 

So to add to the resources already mentioned, the edk2 port is open-source - they did it.

 

Btw, I have also read that uBoot can boot grub directly, but I think you have to supply it with the address to load it into. Without knowing what I'm doing (when I first got my SBC), I didn't want to touch that.

 

Edited 11 hours ago by The Tall Man

[usual user]

Everything you wish for here is achievable with pure U-Boot technology. A long time ago, I tried to practice how to realize this here in the forum, but my lesson from it was:
You can't teach old dogs new tricks, and Armbian users want to celebrate their cargo cult and hope that it will magically fulfill their wishes.
 

[laibsch]

  

9 hours ago, usual user said:

Everything you wish for here is achievable with pure U-Boot technology.

 

Awesome!  Please do elaborate.

  

9 hours ago, usual user said:

You can't teach old dogs new tricks, and Armbian users want to celebrate their cargo cult and hope that it will magically fulfill their wishes.

 

Please do elaborate.  What is the cargo cult practiced here at Armbian?  I'm old, but I'm still interested to learn, so please do teach me.

[usual user]

2 hours ago, laibsch said:

Awesome!  Please do elaborate.

Since the RK3399 U-Boot can use an HDMI display and a USB keyboard, I would simply configure a jumpstart option in the boot flow that mounts a different root filesystem. When booting, you just have to select this option.
If interacting with the firmware console is too complicated, the recovery system can be placed on a removable storage device. In this way, in case of need, only the rescue media needs to be connected and the system restarted; no firmware console access is required.
A completely firmware-controlled fallback mechanism is also possible, but it requires further special configuration of the firmware.

 

2 hours ago, laibsch said:

What is the cargo cult practiced here at Armbian?

Read this thread to understand what I mean by my statement.

[laibsch]

23 minutes ago, usual user said:

Since the RK3399 U-Boot can use an HDMI display and a USB keyboard

 

Thank you for your answer.  My board is not Rockchip-based, we have been generalizing the discussion for quite a while now.

 

24 minutes ago, usual user said:

A completely firmware-controlled fallback mechanism is also possible, but it requires further special configuration of the firmware.

 

 

Well, and this is where it would get interesting but ...

[usual user]

1 hour ago, laibsch said:

My board is not Rockchip-based

Oh, sorry, I didn't notice the non-existent 6 and read it as Helios64.
Of course, my description of the boot method is not limited to Rockchip devices; it works on all for which a mainline U-Boot is available. I have used it on iMX6, LX2160A and S922X devices, but my remaining devices are all based on Rockchip.

 

1 hour ago, laibsch said:

this is where it would get interesting but ...

The solutions are too varied to present a turnkey solution here. However, I am sure that only a corresponding configuration for implementation is required to achieve the desired behavior, but for that, the U-Boot documentation must be consulted to decide which solution should be chosen.