Separate boot partition and different root partitions → simple man’s A+B update possible?

[LanMarc77]

Getting into buildroot and yocto seems a bit too much for me to get an A+B style update possibility. I would like to stick with the quality images provided by Armbian.

The following applies to a NanoPi-NEO3 others SBC might have different partition layouts.

 

From what I understood is that uboot looks for /boot in /dev/mmcblk0p1 reads the boot.cmd/scr and armbianEnv.txt and then boots the kernel accordingly.

The armbianEnv.txt seems to point the kernel to the root partition using the rootdev directive.

 

If /dev/mmcblk0p1 would now only hold /boot would it be possible to point to a different partition for the rootfs using armbianEnv.txt?

If so I could imagine changing the partition layout to the following:

 

/dev/mmcblk0:

/dev/mmcblk0p1: boot partition

/dev/mmcblk0p2: rootfs partitionA

/dev/mmcblk0p3: rootfs partitionB

/dev/mmcblk0pX: multiple more partitions that survive updates

 

Too now allow consistency with the rootfs partitions also the boot partition directory structure would slightly change. It would contain 2 directories: /bootA, /bootB AND a hardlink /boot which either points to /bootA or /bootB

In the running rootfs the fitting /boot is mounted via a bind mount hoping normal apt-get updates can deal with this. This would allow creating a rootfs by updating and verifying everything works locally and then creating an image from it.

 

Now to OTA a relatively simple script running from rootfsA can download a new rootfs image/file structure and place it in the partition of rootfsB. Same with the new boot partition just into /bootB. The values of the armbianEnv.txt from the new boot image always point to the corresponding rootfs partition either because the image was already built for this or the script dynamically adjusts them. Same goes for fstab, machine-id, ssh identity and other rootfs specifics. The script will then verify that the write worked to avoid sd card issues.

The (almost) atomic operation for switching the systems would be changing the hardlink of /boot to the /bootB directory and reboot.

 

This idea will not detect any boot issues and revert automatically back as uboot is not involved. Also this only works if uboot is still compatible with the new boot files provided. But an additional new uboot image could also be provided during the update.

Reverting can be done manually (relatively fast) by changing the hardlink back and rewriting a previously created backup copy of a might be updated uboot image.

 

Independent of the fact that buildroot, yocto, rauc, mender… systems have a better feature set, do you think this could work?

[eselarm]

19 minutes ago, LanMarc77 said:

If /dev/mmcblk0p1 would now only hold /boot would it be possible to point to a different partition for the rootfs using armbianEnv.txt?

That is basically what I have been doing. I reconstructed image/partitions to have bootfs and rootfs separated. Typically format bootfs as simple FAT where you can fix things even on Windows computers maybe when SD-cards. For new fast SBCs I just have many partitions on NVMe where various Linux distros rootfs are and just copy or rename armbianEnv.txt which is then essentially only a UUID of a rootfs.

 

When I know the SBC well enough, it is more the kernel only that matters, so i made a extlinux.conf generator so I can select a kernel via U-Boot serial console.

 

I don't see what A+B brings me as I use Btrfs for rootfs and use snapper to make 'last-known-good' snapshots and also transfer those to NAS or so. If you want totally unattended, then A+B is an option, look at Android as well I would say.

But more towards PC like systems, one can put efi bootloader (efi-grub) on the bootFAT, that allows you to select last-known-good more or less if you deploy snapshots. I usually do update in-place, so more like rolling release, but you can also do updates on a new read-write snapshot and fallback if it would fail to boot, see https://kubic.opensuse.org/documentation/man-pages/transactional-update.8.html  for example how that works.

 

U-Boot also has options I think to get to last-known-good automatically, but I don't know how that would work. For my stuff at home, it is good enough that I can quickly go back to older snapshot and that works for more than a decade on my PCs, so I do the same on SBCs/embedded, as long as they have a serial console cable (or HDMI,keyboard). For the NanoPi-NEO I also had Btrfs capable U-Boot, so then U-Boot can directly boot a certain rootfs partition, but it is not really default, so I keep an extra bootFAT.

 

[LanMarc77]

So am I reading correctly from your post that separating boot and rootfs using armbianEnv.txt does work?

Do you think the hardlink option inside the boot partition could work?

 

I would most likely test this setup but would like to know before if there are issues why it never would work to avoid work,

I know there are many other options including an initramfs with btrfs support to avoid partitions for different rootfs to then use subvolumes. Still I would like to stick for the time being with the original idea.

Thanks.

[eselarm]

13 hours ago, LanMarc77 said:

So am I reading correctly from your post that separating boot and rootfs using armbianEnv.txt does work?

Do you think the hardlink option inside the boot partition could work?

armbianEnv.txt sets parameters for the U-Boot bootloader. In theory you can also manually type it all via serial console at U-Boot prompt. Separating boot and rootfs is independent of that.

 

If you use Ext4 for bootfs, you can make hardlinks (or symlinks). That is also what is done by kernel install script if /boot is on Ext4, else copies are done (when FAT).

Edited August 25 by eselarm

[LanMarc77]

Thanks @eselarm for the input that this should work. So I tested it. And it does!

I leave a few hints for anyone who wants to replicate.

When creating a new image layout working directly with the image files was the fastest for me. So you need to download one original Armbian image. I used a minimal cli image.

 

Image preparations

Create a new image file (e.g. with dd) that will be partitioned like described above.

Copy all the first sectors until the first partition starts from the original image to the new image. This will copy the original partition table but more importantly also the uboot bootloader into the new image (32768 sectors = 16MB).

 

Partitioning

Partition the new image as described above (e.g. with fdisk/sfdisk) and format the new partitions. One smaller boot partition (e.g. 512MB, ext4) and two bigger root partitions (e.g. 3GB, ext4). I will use the terms rootA and rootB for the two new root partitions in the new image from here on. Get the UUIDs of the newly created boot and rootA partition of the new image (e.g. lsblk/blkid).

Most likely instead of using two separate ext4 rootA and rootB partitions one bigger btrfs partition and the usage of subvolumes could be possible. But I did not test this.

 

Copy boot and adjust

Mount the new boot partition and create directories bootA and bootB and a relative symlink boot pointing to bootA. Place all files from the original image’s /boot directory into bootA of the new image. Adjust the UUID line in armbianEnv.txt that still points to the original image’s root partition to the rootA UUID from the new image.

 

Copy root and adjust

Mount rootA and copy all the files from the original image into it. Remove all files from the /boot directory in rootA as we will mount the boot partition into this directory anyway and do not want to get mixed up. Create a new directory /rawBoot in rootA.

Edit the fstab of rootA and change the UUID that still points to the original image’s root partition to the rootA UUID. Add a line to fstab to mount the boot partition of the new image into /rawBoot. Add a line to fstab to bind mount the directory /boot of /rawBoot into /boot of rootA.

Add an empty file /root/.no_rootfs_resize in rootA to avoid the partitions being changed if the original image was never booted before.

 

I did all of the above also with an archived older Armbian image and put that into rootB (and bootB).

With this setup we now have two separated OS versions. They can be switched by simply changing the boot symlink to bootA or bootB and reboot. Because we used bind mount we can also do kernel upgrades which end up in the correct directory.

As all of the steps are easily automatable by a shell script we can now download a new earlier prepared update package place its content into the other boot directory and partition adjust their UUIDs (and maybe copy SSH keys and machine-id if we want to keep system identity) and change a symlink.

If we need to go back to the older image just the symlink needs to changed back. Either from within a running system or if that is not possible by taking the card out and doing this externally.

 

As I said earlier this A+B update scheme is not as robust as other tools. But we can stay with prebuilt images from Armbian that can be locally configured the way we want it. After this an update package can be created and distributed which could also contain a new uboot. If this is tested well before distribution I guess most errors that make the other tools so robust can be avoided.