Jump to content

Recommended Posts

Posted

I'm writing from a limited perspective, so constructive criticism appreciated. In this thread, I'm recording some thoughts about how the OS gets from files into the memory of various boards, and what might make this more unified and easier and/or more flexible.

Current U-boot

My knowledge comes from the mvebu64 and rockchip64 families.  At this time, the board boot process starts from code in the processor that attempts to find u-boot, and u-boot runs a script from its environment that searches for a UEFI application or a boot.scr file.  They call this "Distro-boot"  Armbian uses the boot.scr to load an uncompressed kernel image and a compressed initramdisk, along with a device tree.  These use u-boot legacy image format.  U-boot then passes control to the kernel.

On mvebu64 (espressobin), u-boot is contained within the SPI on the board, along with its environment.  U-boot 2022.04 on this board can do network booting as well as searching emmc, sd, or sata devices for a boot device based on a GPT type or simply the existence of files within the first partition on each device.

On rk3399, some earlier stage attempts to launch idbloader from sector 64, which loads u-boot.itb from sector 16384, from whatever device can be detected: SPI, USB, eMMC, SD.  The boot.scr loads the legacy uboot images from the first partition on the device, then applies overlays - I believe this enables HATs on boards that can use them. 

 

U-boot capabilities

Boot process: UEFI.  U-boot can load the linux kernel through the UEFI stub, or the GRUB bootloader.  As far as I can tell, it's not a drop-in replacement for a legacy kernel boot.

Boot file format: Flattened uImage Tree.  This is a monolithic file that has one or more of kernels, initramdisks, and device trees, along with "configurations" that can be selected to tell u-boot which files to actually load.  Additionally, the components of this file can be compressed, as u-boot can uncompress the kernel before passing control.

Boot media: dhcp(v6) to tftp.  U-boot can attempt to boot an EFI application from tftp, before it attempts to run a script from dhcp, before it attempts to PXE boot.

Extlinux script: u-boot can read /extlinux/extlinux.conf where a menu of kernels can be provided, with paths to kernel, initrd, and dtb, and an entry for the cmdline.  Each kernel on the system would need to contribute a fragment of this config, which isn't currently done.

 

Image Layout

Currently, many armbian images use an MBR partition table.  It's the default, and a device needs to request GPT during the build to get it.  But with GPT come a few advantages.  The systemd/freedesktop people seem to have a vision of a single layout that would allow a linux system to boot on several architectures, and then auto-configure based on "discoverable partitions".  There are partition type uuids for such things as ARM64 root partition, or dm-verity superblock and signature partitions.  These features would give a read-only, integrity checked, signed image.  There are partition flags for things like "read-only" (60), "no auto" (63), or "grow-file-system" (59).  Beyond this, the ESP (EFI System Partition) has a structure to allow multiple kernels/OSes to deconflict the storage of their files. 

 

Boot Sequence

Type #1 Boot Loader Specification:

$BOOT is the (typically) vfat partition with type "bc13c2ff-59e6-4262-a352-b275fd6f7172", or the ESP (VFAT, "c12a7328-f81f-11d2-ba4b-00a0c93ec93b").  Under $BOOT/loader/entries/, there would be files (e.g. ${machine-id}-${uname -r}-${os-release}.conf) that give configuration to find kernel and initrd at $BOOT/${machine-id}/${uname}/{linux,initrd,dtb,dtbo}

Type #2:

Unified kernel image - EFI PE w/ stub loader, initrd, and commandline, and can be signed cryptographically.  These are at $BOOT/EFI/Linux/*.efi

 

I assume these are handled by GRUB, as they do not seem to match u-boot's distro-boot.  Armbian currently uses the boot.scr from u-boot's distroboot sequence to boot off the filesystem where boot.scr was found. 

 

First-boot and nand-sata-install

When armbian builds the image, it does so through chroot and debootstrap.  This results in artifacts like /etc/machine-id or /etc/ssh/ssh_host_*_key, which shouldn't be on multiple systems.  They're removed as part of the image creation.  Partition UUIDs ought to be changed too, but these are less likely to be a problem. 

The nand-sata-install attempts to create partitions to make a filesystem on some other block device, and then rsync's the running armbian system over to the new block device.  It also tries to manage the u-boot bootloader and all the possible ways that can exist.

 

(Goal) SD card partitioning

sgdisk -n 14:${biosstart}:${biosend} -c 14:bios -t 14:ef02    -n 15:${uefistart}:${uefiend} -c 15:efi  -t 15:ef00    -n  1:${rootstart}:0 -c  1:root -t 1:8305 -A 1:set:59 ${SDCARD}.raw

This produces a GPT-formatted disk with the first partition (numbered 14) with a BIOS type (recommended 1 mb, or covering u-boot and files), an ESP (recommended 550 MB, to mount on /efi/), and an auto-detected ARM64 root partition that is labeled to grow to fill the disk when booted.  Linux should be stored in a UEFI or FIT image, preferably as a UEFI boot method.  U-boot environment variables for a given board should be able to select the appropriate armbian kernel, allowing the root filesystem to have multiple kernels for different boards installed and an SD card that boots on multiple boards.

Posted

I see on meson64, write_uboot_platform writes 442 bytes to the beginning of the disk (out of 446 reserved for bootloader code, why the missing 4 bytes?) and then continues to write the rest of uboot.  MBR fits in the 66 bytes that get skipped, while GPT would need to use the next 16 kB (LBA 1-33 with 512b sectors) and that's conflicting with the u-boot code.

 

I wonder if it would be possible to get the uboot stage 1 code in the first sector to jump to 1 MB rather than just over the MBR, perhaps with some setting in the u-boot code.  Better still would be searching for a BIOS type GPT partition, that's what that type is for. 

 

This is a problem, but one that looks like it can be handled at image creation without having to run commands on the amlogic board.  Thanks for the note.

Posted

So, "u-boot.bin" isn't just BL31 u-boot, but is lower levels of bootloader stages too.  And it looks like they may be encrypted to the processor too, so there's no chance of a human-readable "jump" instruction that could be redirected.  Disappointing.

Posted
  On 5/13/2022 at 3:46 AM, ManoftheSea said:

allowing the root filesystem to have multiple kernels for different boards installed and an SD card that boots on multiple boards.

Expand  

I already have this with native mainline uboot without special boot partition or even a boot directory. My SD card boots on several SBCs (NanoPC-T4, ODROID-N2+, Honeycomb, ...) and the several kernels I have installed concurrently are only a reboot and a keypress, to select the desired one, away. Booting different systems (on separate partitions) is also working, but I use it seldom. And the good thing, to configure everything, only a simple text file is needed.
I have already used this boot schema with various Armbian images for some reverse engineering tasks. Armbian had everything necessary available when using mainline uboot, at least with all images I used. It was only a question of creating a suitable configuration file and copy some files around.
If you are eager to experiment, I can try to talk you thru the configuration. For our experiment an SBC with rk3399 and an SD card with an Armbian image which uses mainline uboot would be best suited because I am most familiar with this SOC.

Posted
  On 5/13/2022 at 5:53 PM, ManoftheSea said:

Yeah, tell me about "suitable configuration file and copy files around".

Expand  

As a first step, I need some information about the running system on which we want to do the experiment.
Please post the output of the following commands:

cat /proc/cmdline
ls -hal /boot
lsblk -o +PARTUUID

so I can prepare the experiment.

 

  On 5/13/2022 at 5:53 PM, ManoftheSea said:

How does uboot match the board with the proper kernel to launch?

Expand  

It is the right DTB what really matters.

Posted

I've got access to both the Helios64 (rk3399) and the espressobin (mvebu64).  Both are on 5.15 in "current".  Since the rk3399 loads firmware from the sd card, I expect that would be an easier base from which to work - the ebin loads u-boot from SPI.

Is it part of the boot.scr that looks up the board on which its running, so as to get u-boot to load the proper DTB?  Surely it can't use the symlink to simply "dtb/" but needs to be customized to 5.15.31-mvebu64 or 5.15.31-rockchip64.

 

  Reveal hidden contents

 

Posted (edited)

OK, here is the preperation for our experiment:
First step: prepare kernel package (copy files around)

cd /usr/lib/modules/5.15.31-rockchip64
cp --preserve --recursive /boot/dtb* .
cp --preserve /boot/System* .
cp --preserve /boot/config* .
cp --preserve /boot/Image .
cp --preserve /boot/vmlinuz* .
cp --preserve /boot/initrd* .

Second step: prepare for dristro-boot

mkdir /extlinux

- Copy the attached extlinux.conf into /extlinux/extlinux.conf over.

- Reboot and check if the device is now booting with distro-boot method
The auto boot timeout is set to 10 seconds, so be patient.

Please post the uboot output because I wrote the extlinux.conf without any testing possibility and hopefully did not make any typos.

extlinux.confFetching info...

Edited by usual user
correct cp command
Posted

I forgot about extlinux.  To support that, our kernel packages would need to be able to contribute fragments to the generated extlinux.conf, which probably belongs somewhere in /etc/initramdisk/post-install/ scripts.  I've updated my first post to mention it.  This also seems like a good way to have different kernels even for the same board, such as what I hear about a "media kernel".

Posted

I was able to test the extlinux.conf (though I moved some things around).  I used the existing /boot/Image and /boot/uInitrd symlinks.

The fdtfile name, that comes from the u-boot environment on the device, right?  So while a kernel that targets aarch64 might be able to boot on multiple systems, the u-boot on the system has to know the right name to find the dtb.  And if that changes between kernel versions, then u-boot's environment must be kept current to seek out the fdt and load it.

 

 

Posted
  On 5/13/2022 at 9:35 PM, ManoftheSea said:

though I moved some things around

Expand  

Please do not do this, it will prevent the usability of kernels installed at the same time.
In the end, we won't use anything in /boot.

 

  On 5/13/2022 at 9:35 PM, ManoftheSea said:

the u-boot on the system has to know the right name to find the dtb.

Expand  

Firmware (uboot) is the only device specific code and is dedicated to each SBC. Having it on the SD card prevents it usability for different devices. The best is if you can offload it in a SPI flash. There is not one uboot that suits everyone.

Posted
  On 5/13/2022 at 9:35 PM, ManoftheSea said:

I was able to test the extlinux.conf

Expand  

I am still awaiting the uboot messages log. We are still in the stage of redoing the legacy boot features with distro-boot method. If I have verified that it is working as I expect, I can start teach how to use the now possible new features of distro-boot.

Posted

I did not move the files from /boot to /usr/lib/modules, and I edited the extlinux.conf to point at the files in /boot.  I think I understand what would be accomplished by moving the files, though.  I actually did the test on ebin (mvebu64) with a default uboot environment and saw that it was able to boot the kernel and bring up the system. Do you really need the actual console output? It looked like it just says it's booting the kernel, same as it ever does.

 

As far as ebin goes, I moved that to using FIT images indicated by the boot.scr.uimg. The extlinux.conf can't tell uboot to use this itb file, I don't think.  Which means no compressed kernels and no uefi boots.

 

Why avoiding anything in /boot, though?  I think the desired state (which has been pointed out that it has conflicts with bootloader code) would be an ESP that has files from any number of OS's or distributions and one or more linux partitions for the Armbian system, identified through the root=UUID=xyz (or autodiscovered), which don't conflict with other distributions.  But uboot would need to save efivars to point at files and cmdlines.  If the Armbian image has a generic aarch64 kernel and dtbs for multiple boards, and uboot can pick the specific dtb to be used, wouldn't this allow for a universal Armbian image that can coexist with other OS's?

Posted
  On 5/14/2022 at 5:24 PM, ManoftheSea said:

did not move the files from /boot to /usr/lib/modules

Expand  

II never asked for, the cp command does a copy, i.e. the source file keeps in place. We only add files to the root filesystem and the lagacy support stays in place. You can always cancel our experiment by renaming extlinux conf, e.g.

mv extlinux.conf extlinux.conf-disabled

 You are then back to your old method with some more files in the root file system. Of course, you can also delete all added files to remove them completely.

 

  On 5/14/2022 at 5:24 PM, ManoftheSea said:

I think I understand what would be accomplished by moving the files

Expand  

With your wrong assumptions you come to wrong conclusions.
Please let's do the experiment as I suggest, and you'll ask your questions at the end when you know the whole story.

 

  On 5/14/2022 at 5:24 PM, ManoftheSea said:

I actually did the test on ebin (mvebu64)

Expand  

Please use the Helios64 for the experiment. I don't have any of your devices and using a well-known SOC is already complicated when I have to rely on your eyes and hands. We can come back to espressobin if we were successful with Helios64.

 

  On 5/14/2022 at 5:24 PM, ManoftheSea said:

Do you really need the actual console output?

Expand  

When your boot process happens as always it sounds to me as if it has fallen back to the legacy method. It would therefore be important for me to know the exact messages.
Oh, I forgot to ask, does your current uboot support the USB keyboard so you can interrupt autoboot and make keyboard inputs? If not, we need the serial console for now and take care of uboot USB keyboard support afterwards.

 

  On 5/14/2022 at 5:24 PM, ManoftheSea said:

Which means no compressed kernels and no uefi boots.

Expand  

The Image link point to vmlinuz and that is a compressed kernel and we are using
distro-boot and that has nothing to do with the uefi methode.

 

  On 5/14/2022 at 5:24 PM, ManoftheSea said:

Why avoiding anything in /boot, though?

Expand  

By the end of our experiment, it will be obvious to you.

 

  On 5/14/2022 at 5:24 PM, ManoftheSea said:

But uboot would need to save efivars to point at files and cmdlines.  If the Armbian image has a generic aarch64 kernel and dtbs for multiple boards, and uboot can pick the specific dtb to be used, wouldn't this allow for a universal Armbian image that can coexist with other OS's?

Expand  

By the end of our experiment, you will know how it works.

Posted

Do you really have to be so mysterious?

 

vmlinuz is NOT compressed on aarch64.  The FIT image allows telling u-boot that the kernel image is compressed, and uboot does the decompression, not the kernel.

 

Here's uboot log:

  Reveal hidden contents

Here's /extlinux/extlinux.conf

  Reveal hidden contents

Here's /boot/

  Reveal hidden contents

I'm using serial console in the case of the ebin, and I will use serial console for helios64.  That'll be a little slower to run, I'll get to it.

 

 

Posted

Here's the Helios64:

I had to add
`cp --preserve --recursive /boot/dtb-5.15.31-rockchip64`
to the instructions given, as the first command only copied a symlink

Directories:

  Reveal hidden contents


/extlinux/extlinux.conf - slight differences

  Reveal hidden contents


uboot log:

  Reveal hidden contents

 

Addl info:

The espressobin is using uboot 2022.04, while the helios64 self-identifies as "ver=U-Boot 2020.10-armbian (Sep 08 2021 - 10:55:50 +0000)".  It is interesting to me that the helios u-boot has more uboot variables than the espressobin.  The serial# and ethaddr are bogus, coming from the Armbian image because of uboot variables on the sd card, right?

 

Compression:

  Reveal hidden contents

 

Posted
  On 5/14/2022 at 9:57 PM, ManoftheSea said:

Do you really have to be so mysterious?

Expand  

I am still in the phase to learn what your system exactly provides. I know what to look for, since I don't know the results before, I can't deliver a turnkey system. I therefore have to check individual points step by step and, if necessary, establish appropriate workarounds. Therefore, it is necessary to follow my instructions as unmodified as possible.

 

  On 5/14/2022 at 9:57 PM, ManoftheSea said:

vmlinuz is NOT compressed on aarch64

Expand  

For me it is:

file vmlinuz
vmlinuz: gzip compressed data, max compression, from Unix, original size modulo 2^32 46146048

A point that kann be tackled for Armbina afterwards.

 

  On 5/15/2022 at 3:32 AM, ManoftheSea said:

I had to add
`cp --preserve --recursive /boot/dtb-5.15.31-rockchip64`

Expand  

Well caught, you have applied the right solution, link and directory are needed.

 

  On 5/15/2022 at 3:32 AM, ManoftheSea said:

Here's the Helios64:

Expand  

The logs are already very promising, but do not yet confirm all the required points. I need the outputs with my unmodified extlinux.conf as I attached it. Please repeat the process by removing the link /usr/lib/modules/linux, restore my extlinux.conf, restart the device, wait for the autoboot timeing out and post the log after the boot is complete.
If everything is as I expect it to be, there is only one imponderable to check and we can finally start our distro-boot exercise.

Posted
  On 5/15/2022 at 6:21 AM, usual user said:

For me it is

Expand  

How?  This isn't a kernel configuration option, so what is doing so afterward?  And then how are you telling uboot to decompress it, that's not in your extlinux config?

 

https://www.kernel.org/doc/Documentation/arm64/booting.txt

"The AArch64 kernel does not currently provide a decompressor and therefore requires decompression (gzip etc.) to be performed by the boot loader if a compressed Image target (e.g. Image.gz) is used. For bootloaders that do not implement this requirement, the uncompressed Image target is available instead."

 

I will remove the /usr/lib/modules/linux symlink.  But I'm still going to leave the extra /usr/lib/modules/5.15.31-rockchip64/boot/ directory.  It looks like you expect the default option to fail and you want to see if it will fallthrough.

 

 

Posted

Bottom of log, default with timeout:

  Reveal hidden contents

 

Extlinux.conf

  Reveal hidden contents

 

Posted
  On 5/16/2022 at 12:46 PM, ManoftheSea said:

It looks like you expect the default option to fail and you want to see if it will fallthrough.

Expand  

Exactly.

 

  On 5/16/2022 at 12:46 PM, ManoftheSea said:

But I'm still going to leave the extra /usr/lib/modules/5.15.31-rockchip64/boot/ directory.

Expand  

Again you have modified my suggestions, so slowly I lose the desire to continue this training.
A boot directory is a legacy boot artefact that isn't needed for distro-boot.
Either you want to learn from me my distro-boot deployment and follow my suggestions, or we leave it and I save myself a lot of frustration.
Since I am convinced from the previous observations that your system has the necessary prerequisites, here is a last attempt to continue the training.
Use my unmodified extlinux.conf and use the locations that evoke these commands:

cd /usr/lib/modules/5.15.31-rockchip64
cp --preserve --recursive /boot/dtb* .
cp --preserve /boot/System* .
cp --preserve /boot/config* .
cp --preserve /boot/Image .
cp --preserve /boot/vmlinuz* .
cp --preserve /boot/initrd* .

After the training, you can then make all the changes you want when you implement it for Armbian.
As a next step, we will now install another kernel. Since I can assume that you do not have a suitable kernel package available, I have uploaded my kernel, as I currently use it on all my devices, here. It is a pure mainline kernel built as a generic kernel, i.e. it should be suitable for all aarch64 devices that have corresponding mainline support. Helios64 support is built, but I'm not sure if everything necessary is included, but the backup 5.15.31-rockchip64 is in place to keep your System usable.
Unpack it with this command in your /usr/lib/modules/ branch:

tar -C /usr/lib/modules/ -xzf 5.18.0-0.rc3.27.fc34.aarch64.tgz

This will populte a new kernel package branch in /usr/lib/modules/ and aktivate it for the default boot option.
Reboot your Helios64 and report if the new kernel fully starts.

Posted

So, how's fedora compressing the vmlinuz?  How is uboot recognizing it and unpacking it?  Can I just point uboot at any old compressed kernel?

 

  Reveal hidden contents

So a successful boot of the fedora kernel.  Full log attached.

 

I also attempted the 5.16 kernel from debian, but it did not find the console or network, so I can't claim it ever successfully booted.

minicom-iron.capFetching info...

Posted

Congratulations, now we are in business.

  On 5/16/2022 at 9:01 PM, ManoftheSea said:

So, how's fedora compressing the vmlinuz?  How is uboot recognizing it and unpacking it? 

Expand  

It uses exactly what is described in your quote (https://www.kernel.org/doc/Documentation/arm64/booting.txt). The kernel is build with the Image.gz target and the firmware (uboot) decompresses it. And, surprise, uboot can automatically detect at what kind of kernel the extelinux.conf kernel key points. I have so much more to elaborate about what has been achieved so far, but no time right now. So I have to put you off on a follow-up post, please stay tuned.

Posted

I've found the debian package "u-boot-menu" which exists specifically to read existing kernels and generate /boot/extlinux/extlinux.conf.  It stomps all over any other OS configuration while doing so.  Hmm.

Posted

First of all, a summary of what we have achieved.
By confirming that my kernel package works we now have an Armbian system which boots on a Helios64, a NanoPC-T4, an Odroid N2+, an Odroid N2, and a Honycomb in the same way. I have the certainty that my fedora system works without modifications on a Helios64. And the only requirement to be fulfilled for this is that there is a suitable firmware (uboot) in the required place. Wherever this is (SPI flash, eMMC, SD card, ...). The extent to which the user space configuration has to be adapted is another story. For fedora I am sure that it works.
You already know how to install a kernel package and enable it for the default boot option. For a kernel update you only need to ask me to provide the next version and can then install it with an equivalent tar command. If something goes wrong (e.g. a component is missing), you have seen the distro-boot try the next boot option. As long as distro-boot is in control, this will continue until all boot options have been tried. If the control has already been passed to the kernel and it stucks, it is necessary to select an "known good" option with the keyboard while the autoboot timeout has not yet expired. As you can see, you have to be relatively creative here to have a system that can't be started somehow.
Now let's include a little convenience feature. We want to keep easy access to a previously installed kernel. To do this, we duplicate the first boot options entry, and replace the linunx path part with linux-previous everywhere. Now it is enough to run

mv /usr/lib/modules/linux/ /usr/lib/modules/linux-previous/

before a kernel update to support the new boot option. If we want to keep a kernel permanently accessible via boot option, we replace the linux part with the real kernel branch part.
But now you want to use Armbian kernel primarily. To do this, we duplicate the last boot options entry, and replace the real kernel branch part with e.g. linux-armbian everywhere. In the attached extlinux.conf I omitted the initrd line. So you can check if the Armbian kernel also works without initramfs and you can save the loading in the future. Before the new boot options work you have to create the new management links pointing to 5.15.31-rockchip64.
Through different management link name bases, you can easily manage several kernel variants.
To use the Armbian kernel as default, you could move the corresponding boot option in the first place, but there is the more elegant default distro-boot key. This is used to specify which boot option to start the boot trys with.
As you have probably already noticed, the append lines differ between fedora and Armbian kernel. And even for different SOCs, different parameters may be required, e.g. serial device name. Also, parameters cannot be portable if kernels are built with other configurations. You therefore have to pay attention which kernel packages you use with which append lines.
And now a few words about kernel packages. As you may have noticed, my package comes without an initramfs. This is possible because my kernel is built in such a way that all drivers necessary to access the rootfs are build-in and was my biggest concern that something would be missing for Helios64. Fortunately, this is not the case. The initramfs is always rootfs dependent, so it cannot be easily exchanged between different systems. It must always be generated in the current system. It is therefore good practice to build the kernel in such a way that no initramfs is needed to have a portable kernel package that can be used for each rootfs for a specific SBC, e.g. fedora kernel usable in Armbian.
Oh, by the way, please try not to test multimedia components, it could be that you notice that they are technically up to 5.19.0+ ;-). But I'm afraid that your userspace components are not up-to-date enough. It requires at least the latest mesa and gstreamer releases and FFMPEG requires out of tree patches because mainline support is missing.
And now a little digression, transfer your SD card to the ESPRESSOBin and post the log created with the default verbose boot option.I'm sure my kernel package won't start completely, but I'd still like to see how far it gets.

 

  On 5/17/2022 at 4:15 PM, ManoftheSea said:

I've found the debian package "u-boot-menu"

Expand  

We'll look at this again when you know all the complexity of extlinux.conf, there's still something to come.
So now I have to think about how to continue the training, there are still so many aspects to explain.

extlinux.confFetching info...

Posted

In this exercise, we will work on two rootfs. To show that the distribution used is not important, I will use my fedora image for demonstration.

 

First step, setup:
Create a second partition on the same storage device. Choose a size as you want to use it in later use, but at least so large that the selected rootfs fits into it. Gain access to the desired image. Either via installed media or via loop-mount. Use dd to transfer the rootfs partition to the partition you created earlier. Resize (resize2fs) the copied file system to match the partition size.

 

Second step, configuration:
My fedora image already uses distro-boot, but the extlinux.conf needs a little adjustment. So mount the second partition (/mnt/second) and edit /mnt/second/extlinux/extlinux.conf. Update all PARTUUID values by the actual one. If you have used the second primary MBR partition entry this would be 8b029f90-02. But for safety you should check it (lsblk -o +PARTUUID).
If you use a rootfs without distro-boot support, copy the current extlinux.conf, but also adjust all PARTUUID values. Also copy all relevant kernel packages from /usr/lib/modules. Make sure that they are portable kernel packages, i.e. do not use an intrd line (initramfs). If you want to use the original kernel, you should already know what to do. Prepare kernel package in /mnt/second/usr/lib/modules/${original-kernel}/ and create boot option in extlinux.conf. You should note that an append line matching the original kernel is to be used (cat /proc/cmdline running in the original system or extracted from the legacy boot artifacts).

 

Third step, use:
In standalone mode, the systems are used alternately and independently. To do this, set the MBR boot flag with your preferred partition manager (e.g. fdisk "a" command). The boot flag directs distro-boot in which partition to look for the extlinux.conf. If none is set, the first partition is chosen.


In combined mode, dependencies of the systems involved arise, but you can start them from a single boot menu. To do this, the relevant unmodified extlinux.conf boot options from the secondary system are apended to the extlinux.conf of the distro-boot boot partition. The distro-boot boot partition is the one in which the used extlinux.conf is found. The dependency that arises during use is the necessity that the kernel packages used must be present both in the boot partition and the rootfs of the used system. They must therefore be copied over, and in the case of updates, attention must be paid to appropriate duplication. This is necessary because disto-boot expects kernel artifacts in the partition where extlinux.conf was found and a booting system expects kernel modules in the mounted root-file-system.
The extlinux.conf and the extlinux directory in the secondary systems rootfs are not required for this mode and do not need to exist.

 

And now the whole thing when the systems are on different drives.
Now it gets complicated, it works the same as before only now the first component of the PARTUUID is different 😉
The standalone mode may not be possible. It only works if distro-boot also scans the corresponding drive for extlinux.conf. To bypass a drive in front of it in the scan priority list, deactivate the distro-boot there by renaming the extlinux.conf (extlinux.conf-disabled). Of course, you have to also deactivate a legacy boot there by renaming the corresponding script file.
The combined mode works always. It's like legacy boot with a separate boot partition, but boot and system partition don't even need to reside on the same drive.

 

If you want to do a real experiment, take a completely empty SD card (dd if=/dev/zero of=/dev/${to-clean-sd-card}), and copy only the firmware (uboot) artifacts starting at sectors 64 and 16384 to it. Now insert your Armbian system SD card into a USB card reader and then insert it into a USB port on your HELIOS64. When the previously prepared firmware SD card is in the SD card slot it should be booting your system.

Posted

Why are you using PARTUUID instead of filesystem UUID? 

 

I'm not really interested in continuing to use MBR partitions at all.  I don't want to be mounting another distro's root FS to modify boot configuration, that stuff belongs in an ESP.  The kernel, initrd, and cmdline ought to be placed and signed by the given distro.

 

So for most boards, it seems like the best situation would be to have an SPI or eMMC with a boot0 where it the board can place the bootloader code, and then to have an eMMC space to use an ESP upon.  Any further boot artifacts would be placed in that filesystem which is shared between any OSes using the system.  A single file that contains its own initrd (if necessary) and cmdline would be the easiest for other, brain-dead OSes to not mess up.  And possibly the dtb, which ought to be associated with the board and not the running OS, but that doesn't seem to be always true.  Then, it's the initrd and cmdline that looks for other filesystems on other devices, and not the responsibility of the bootloader.

 

I wonder if u-boot can support the "include" keyword.  It'd be easier to have an overarching extlinux.conf that includes distro-specific lists of boot files, than to expect my debian, his fedora, and that other guy's LFS to sort things out in the same text.  Something like:

ESP:/extlinux/extlinux.conf:

INCLUDE /mydistro/extlinux.inc
INCLUDE /fedora/fedora.cfg

ESP:/mydistro/extlinux.inc:

LABEL blah
 KERNEL /mydistro/2022.05/mykernel
 FDTDIR /mydistro/2022.05/dtb
 APPEND whatever configuration is needed

LABEL recovery
 KERNEL ...
 FDTDIR ...
 APPEND ... single

A given OS only needs to check for the presence of a line "INCLUDE $distro" without any further consideration of order, which is up to the user.  It would be allowed to change its own files under ESP:/mydistro/

 

On the other side of this problem, getting a given system to recognize the presence of a remove-able device... hmm, can a detected EXTLINUX configuration have an option to not load anything, and u-boot can go on to trying the next boot device?  That would allow the eMMC to have the system's list of OSes, but also have a fall-through to detect a remove-able extlinux for portability.

 

Or, we need something like PCs have with "press f-12 to select media to scan for extlinux.conf".

Posted
  On 5/18/2022 at 9:23 PM, ManoftheSea said:

Why are you using PARTUUID instead of filesystem UUID? 

Expand  

Because I work according to the KISS and DRY principle.

The kernel can determine the position of the root file system with the help of the PARTUUID without the help of the user space. UUID always requires user space support because there is no native support for this and thus always an initramfs. The knowledge of which drivers are needed for a particular device must be available in any case in order to take them into account in the kernel build. If they are build as modules, you need to make sure that you set up an initramfs, that they are included in the initramfs, that initramfs is provided to the firmware and that the firmware loads it. So many additional possibilities where something can go wrong. If the drivers are build-in, all administrative overhead for the initramfs is natively included in the kernel config. This results in a less error-prone system and takes maintenace burden down. KISS
Once the rootfs is mounted, the kernel can load any module, i.e. only the drivers for rootfs access are required as build-in. All others such as e.g. GPU, VPU network, ... can be modules. Without initramfs they are only available once in the system (/usr/lib/module/5.18....). DRY

  On 5/18/2022 at 9:23 PM, ManoftheSea said:

I'm not really interested in continuing to use MBR partitions at all.

Expand  

I don't really care, but if you insist on using GPT, use GPT PARTUUIDs and my implementation works equally. But good luck if the partition table collide with the areas where the maskrom code requires its firmware (e.g. amlogic) and no seperate firmware storage is available.

 

  On 5/18/2022 at 9:23 PM, ManoftheSea said:

I don't want to be mounting another distro's root FS to modify boot configuration

Expand  

This is only necessary if you colocate several systems, in my implementation each rootfs corresponds to a ESP. As soon as distro-boot finds /extlinux/extlinux.conf, this system will be started accordingly. At the moment a fixed scan sequence of uboot is predetermined, but it is already being worked on mainline uboot to scan all drives and a boot menu with all found systems will be offered for selection. Keyword "bootflow", and also UEFI systems are to be supported.

 

  On 5/18/2022 at 9:23 PM, ManoftheSea said:

So for most boards, it seems like the best situation would be to have an SPI or eMMC with a boot0 where it the board can place the bootloader code

Expand  

This is a pious wish and I hope you will be heard by the board designers. I would be happy if they would offer always mainline uboot instead of bitrotten outdated BSP hacks. I could handle the rest.

 

  On 5/18/2022 at 9:23 PM, ManoftheSea said:

Any further boot artifacts would be placed in that filesystem which is shared between any OSes using the system.

Expand  

But good luck convincing all distributions to provide compatible configurations. I prefer the way mainline uboot goes, each distribution uses its own configuration and uboot uses distro-boot, uefi-boot, lecacy boot.scr, ... whatever it finds.

 

  On 5/18/2022 at 9:23 PM, ManoftheSea said:

I wonder if u-boot can support the "include" keyword.

Expand  

The configuration structure is borrowed from the syslinux  project.This was done so that distributions can continue to use their existing tools, only some keywords for the devicetree treatment were added because they were not necessary in the x86 world. But good luck convincing the distributions to take over your desired extensions.

 

  On 5/18/2022 at 9:23 PM, ManoftheSea said:

can a detected EXTLINUX configuration have an option to not load anything, and u-boot can go on to trying the next boot device?

Expand  

Not that I know, so my rename extlinux.conf solution. And even then it falls back on legacy boot, uboot exhausts all possibilities to start the system somehow. Multisystem configurations are not the standard in the embedded world.

And now the absolute killer argument: When can I use your planned solution out-of-the-box for each distribution. Mine has been working productively for years. And I can switch other distributions to distro-boot by just adding an /extlinux/extlinux.conf, as we proved with our experiment because everything else necessary is obvious already available. All other measures where only there to show what new possibilities arise with distro-boot. Or can you explain to me, for example, how to install a second Armbian kernel in your system with the legacy boot system and select at boot time. During the experiment I have already identified some points that can better support distro-boot due to small modifications, but do not affect legacy boot.
E.g. the kernel package:

I let you put some files from the boot directory to /usr/lib/modules/5.15.31-rockchip64, which you eventually did.This served to show you how I am used to how fedora binary kernel packages create these directories. The downstream installation routine then copies necessary components into a possible /boot or ESP directory depending on the boot method used. The kernel binary package is therefore generic and can be used for different boot methods like syslinux, extlinux, grub, uefi, ... , only the install routine decides the details where to put the boot artifacts. DRY
If you now modify the Armbian kernel deb package accordingly, the kernel package is already ready for multiple kernel installations and if your ESP is available only the install routine must be added appropriately. And of course disto-boot can use it out-of-the-box for multiple simultaneous kernel installations.

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...

Important Information

Terms of Use - Privacy Policy - Guidelines