ManoftheSea

I'm writing from a limited perspective, so constructive criticism appreciated. In this thread, I'm recording some thoughts about how the OS gets from files into the memory of various boards, and what might make this more unified and easier and/or more flexible.
Current U-boot
My knowledge comes from the mvebu64 and rockchip64 families.  At this time, the board boot process starts from code in the processor that attempts to find u-boot, and u-boot runs a script from its environment that searches for a UEFI application or a boot.scr file.  They call this "Distro-boot"  Armbian uses the boot.scr to load an uncompressed kernel image and a compressed initramdisk, along with a device tree.  These use u-boot legacy image format.  U-boot then passes control to the kernel.
On mvebu64 (espressobin), u-boot is contained within the SPI on the board, along with its environment.  U-boot 2022.04 on this board can do network booting as well as searching emmc, sd, or sata devices for a boot device based on a GPT type or simply the existence of files within the first partition on each device.
On rk3399, some earlier stage attempts to launch idbloader from sector 64, which loads u-boot.itb from sector 16384, from whatever device can be detected: SPI, USB, eMMC, SD.  The boot.scr loads the legacy uboot images from the first partition on the device, then applies overlays - I believe this enables HATs on boards that can use them. 
 
U-boot capabilities
Boot process: UEFI.  U-boot can load the linux kernel through the UEFI stub, or the GRUB bootloader.  As far as I can tell, it's not a drop-in replacement for a legacy kernel boot.
Boot file format: Flattened uImage Tree.  This is a monolithic file that has one or more of kernels, initramdisks, and device trees, along with "configurations" that can be selected to tell u-boot which files to actually load.  Additionally, the components of this file can be compressed, as u-boot can uncompress the kernel before passing control.
Boot media: dhcp(v6) to tftp.  U-boot can attempt to boot an EFI application from tftp, before it attempts to run a script from dhcp, before it attempts to PXE boot.
Extlinux script: u-boot can read /extlinux/extlinux.conf where a menu of kernels can be provided, with paths to kernel, initrd, and dtb, and an entry for the cmdline.  Each kernel on the system would need to contribute a fragment of this config, which isn't currently done.
 
Image Layout
Currently, many armbian images use an MBR partition table.  It's the default, and a device needs to request GPT during the build to get it.  But with GPT come a few advantages.  The systemd/freedesktop people seem to have a vision of a single layout that would allow a linux system to boot on several architectures, and then auto-configure based on "discoverable partitions".  There are partition type uuids for such things as ARM64 root partition, or dm-verity superblock and signature partitions.  These features would give a read-only, integrity checked, signed image.  There are partition flags for things like "read-only" (60), "no auto" (63), or "grow-file-system" (59).  Beyond this, the ESP (EFI System Partition) has a structure to allow multiple kernels/OSes to deconflict the storage of their files. 
 
Boot Sequence
Type #1 Boot Loader Specification:
$BOOT is the (typically) vfat partition with type "bc13c2ff-59e6-4262-a352-b275fd6f7172", or the ESP (VFAT, "c12a7328-f81f-11d2-ba4b-00a0c93ec93b").  Under $BOOT/loader/entries/, there would be files (e.g. ${machine-id}-${uname -r}-${os-release}.conf) that give configuration to find kernel and initrd at $BOOT/${machine-id}/${uname}/{linux,initrd,dtb,dtbo}
Type #2:
Unified kernel image - EFI PE w/ stub loader, initrd, and commandline, and can be signed cryptographically.  These are at $BOOT/EFI/Linux/*.efi
 
I assume these are handled by GRUB, as they do not seem to match u-boot's distro-boot.  Armbian currently uses the boot.scr from u-boot's distroboot sequence to boot off the filesystem where boot.scr was found. 
 
First-boot and nand-sata-install
When armbian builds the image, it does so through chroot and debootstrap.  This results in artifacts like /etc/machine-id or /etc/ssh/ssh_host_*_key, which shouldn't be on multiple systems.  They're removed as part of the image creation.  Partition UUIDs ought to be changed too, but these are less likely to be a problem. 
The nand-sata-install attempts to create partitions to make a filesystem on some other block device, and then rsync's the running armbian system over to the new block device.  It also tries to manage the u-boot bootloader and all the possible ways that can exist.
 
(Goal) SD card partitioning
sgdisk -n 14:${biosstart}:${biosend} -c 14:bios -t 14:ef02    -n 15:${uefistart}:${uefiend} -c 15:efi  -t 15:ef00    -n  1:${rootstart}:0 -c  1:root -t 1:8305 -A 1:set:59 ${SDCARD}.raw
This produces a GPT-formatted disk with the first partition (numbered 14) with a BIOS type (recommended 1 mb, or covering u-boot and files), an ESP (recommended 550 MB, to mount on /efi/), and an auto-detected ARM64 root partition that is labeled to grow to fill the disk when booted.  Linux should be stored in a UEFI or FIT image, preferably as a UEFI boot method.  U-boot environment variables for a given board should be able to select the appropriate armbian kernel, allowing the root filesystem to have multiple kernels for different boards installed and an SD card that boots on multiple boards.

Ensign's Log, stardate... uh, today.
 
I've contributed changes that build the FIT image as part of /etc/initramfs/post-update.d/99-uboot-fit, which means it'll be updated when new kernels are installed.
 
I've also contributed changes that modify the boot.scr.uimg to use the FIT image rather than the legacy images, and which allows setting "board_version" in armbianEnv.txt to identify which FDT to use from the FIT image.  Options are "v5" and "v7", and the only known difference is the ordering of the ethernet ports.

System successfully upgraded. No problems detected (so far).
Thanks to the team.

Also, the "new bootscript" is no longer required with u-boot 2022.04.  They include the distro_boot method which finds the boot.scr, and that loads the kernel. 
 
env default -a
saveenv

Now I would also like to dream:
When commissioning a new system, I imagine that I will be offered a menu at the first boot in which all configurations available for selection are displayed. Since the SBC can't read my mind, at least I hope so, I choose the one I want. The system then starts and performs its first boot routine. At the next start of the system I wish for a dedicated menu of my previously selected boot option.
 
I'll leave it up to you to explore what changes you think are necessary.
 
Nevertheless, let's see how I would approach the matter in my dream. First, I would create a numbered extlinux.conf for each configuration selection with the following naming scheme: extlinux.conf-xx  with xx=01, 02, ...
The initial extlinux.conf contains only the fail save boot options of the available configurations. In addition, I append a boot parameter of the following form to each append line: bootoption=xx with xx=01, 02, ...
Then I write a shell script that searches /proc/cmdline for bootoption= and then copies the corresponding extlinux.conf-xx to extlinux.conf depending on the value found. Now I make sure that the script is executed as part of the first boot routine, and my dream should come true.
Wait, do I have just developed a method how distro-boot can communicate to the running system which boot option for the successful boot has been used. Am I still dreaming now or is all this already really feasible?

22.05.1_espressobin_bullseye_current_5.15.35.img.xz - no issues on ebin v5
22.05.1_espressobin_jammy_edge_5.17.5.img.xz - no issues on ebin v5

Congratulations, now we are in business.
It uses exactly what is described in your quote (https://www.kernel.org/doc/Documentation/arm64/booting.txt). The kernel is build with the Image.gz target and the firmware (uboot) decompresses it. And, surprise, uboot can automatically detect at what kind of kernel the extelinux.conf kernel key points. I have so much more to elaborate about what has been achieved so far, but no time right now. So I have to put you off on a follow-up post, please stay tuned.

Please do not do this, it will prevent the usability of kernels installed at the same time.
In the end, we won't use anything in /boot.
 
Firmware (uboot) is the only device specific code and is dedicated to each SBC. Having it on the SD card prevents it usability for different devices. The best is if you can offload it in a SPI flash. There is not one uboot that suits everyone.

Hello, @StarSurfer,
This is my fault, and something I'll need to repair for release.  I believe the fastest solution for you would be to mount your image and rename the file boot.scr.uimg to boot.scr .   This won't work because the boot.cmd was changed to use the newer u-boot default parameters.  Follow the directions below.
 
Alternately, if you flash the latest u-boot (2022.04) and wipe the environment variables according to instructions I haven't posted to the espressobin page yet, the u-boot 2022.04 default boot looks for the boot.scr.uimg file.
An initial pass of those instructions looks like, from the u-boot serial console:
- env default -a
- setenv ethaddr <YOUR_ADDR_HERE>
- saveenv

Since we build our own kernel instead of using Debian's upstream, sometimes the kernels are tuned to the device, and sometimes the kernels are detuned through mistake or optimization.  For instance, my own kernel builds have turned off a LOT of device drivers, it's the difference between a 180 minute build and a 25 minute build.
If you would like the Armbian builder to build modules that you use, please submit a PR against https://github.com/armbian/build/blob/master/config/kernel/linux-mvebu64-current.config or linux-mvebu64-edge.config.

There are Espressobin variants with just 512MB of memory. So if your image needs to be compatible with more variants, ensure that you place all parts in first 512 MB of RAM. I'm not sure where is running U-Boot after relocation and where it is malloc area. So choose address which does not conflict with it.
 
You can look at RAM layout used on A3720 CPU:
https://git.trustedfirmware.org/TF-A/trusted-firmware-a.git/tree/plat/marvell/armada/a3k/common/include/platform_def.h?h=v2.6

USB 3.0 is 5Gbps, PCIe 2.1 is 5Gbps and SATA is 6Gbps.
 
For loading you should use some unused space in RAM. Default value of of those variables should contain appropriate values. And they may change between releases, that is why addresses are in variables.

I have tested it on two different A3720 devices and on both it worked. And yes, it is clean U-Boot 2022.04 without any additional patches. So really do not know what can trigger this issue. There can be memory corruption or any other issue which could depend on linker layout, memory usage, etc... But as I cannot reproduce it, it is really hard for me to debug.

Well...  In that case I have a space 32Gb SD card. I shall clone the working SD card, try and upgrade on the copy.  If it works, great.  If it does not, well....   Boo!!!
I shall report back here with my findings.

https://github.com/armbian/build/pull/3642

Sounds like a great use of available features.

@ManoftheSea Yes, as mentioned in my previous post, I came to the same conclusion. It seems that v5 cannot do it, but v7 can. I have already bought the v7 and now I am trying to get it to download the bootloader from from SD card. So far no luck.
 
Also you are correct, my links are not applicable to the problem. However, I have no better links. I am unable to find any link with a description of how to download the bootoader from SD card, be it with espressobin v5 or with v7.
 
The depictions and tables from the link you posted (http://wiki.espressobin.net/tiki-index.php?page=Ports+and+Interfaces#Boot_selection) also seems to be weird. If you look at the table for v7, all the boot modes (other than NOR)  seem to have the same jumper setting. This cannot be correct. Also the order of jumpers for v7 is wrong in the table. The order on the real board is J3, J11, J10, which is quite irritating when using the table to check your settings. This goes further, the depictions and tables for the v5 are flat out wrong. The table says that "NOR boot" is J11=1, J3=0, J10=0, but to get it to boot from NOR flash I actually have to sett it to J11=0, J3=0, J10=1. This sucks and took some time to figure out.
 
Long story short: I cannot find any source with a description of how to download the bootloader from SD card with espressobin v7. Maybe I should open a new thread for that.
 
 
 

Fair enough.  It depends on how I configure the routing, to be sure.  This is going to be much more capable than a raspberry PI in a like configuration which was the other likely candidate.  While I could configure a PI4 as a single-armed router, the EB design provides for more direct routing configuration.

Yes, like @ManoftheSea pointed out above, we are aware that the three onboard ports have a 1gig bottleneck at the SoC. However, with a 1gbps mPCIE card in addition to one of the built-int ports, there is a theoretical 2gbps path. I was only testing the lan0 + lan1 etherchannel group with the sole purpose of trying to get any of the built-in ports to show up in an etherchannel/bond interface (unsuccessfully, so far).
 
@lanefu, I appreciate your experience and insight, and I'd say you're right-on with your assessment of "typical" traffic going to a home fileserver. You have a good point to level-set for anyone coming across this thread, and this most certainly wouldn't be a great option for a HTPC or home media server. From my perspective, however, the Espressobin has a [theoretical] 6gbps path to SATA, but it's a shame the network bottlenecks to 1gbps. In my hypothetical use-case, there would be several hosts simultaneously accessing data on this server. Each of those hosts would have a single 1gig NIC, so of course there would be no point in going above that speed for any single connection. However, as more and more hosts try to access this share a the same time, a handful of 1gig clients could easily exceed a single interface, so an etherchannel would make sense.
 
Also, my experience comes from a work environment where a single link is a potential single point of failure. We don't install anything without redundant links, and an etherchannel is a great way to allow for automated failover in the even of one link failing without having to run some additional heartbeat software. I realize that's probably not something your typical home user is concerned with, but IMO it would be cool to have.
 
One final note along the lines of @lanefu's post, I should mention that in all my tests so far with dual 1gig links to the Espressobin, I am hitting a CPU bottleneck before ever getting close to 2gbps network speed. Obviously I don't have an etherchannel working yet (and I really don't know how that will affect CPU utilization for network throughput), but with one of the built-in ports on VLAN A, and the mPCIE port on VLAN B, using four other test boxes (2 on VLAN B, 2 on VLAN A), all doing reads, writes, or simultaneous reads and writes, I have been unable to achieve much over 100MB/sec total before the Espressobin cores both peg at 100%. I've done a few tests with NFS exports, and a few with NBD exports, but all my tests so far have been limited by the CPU on the server. I'll continue to test and tweak my setup, but at this point, I'm not sure this is the right platform for a high speed NAS server.
 
All that said, I'm still trying to figure out how to add one of the built-in ports to a bond interface... Any suggestions would be greatly appreciated!
 
Edit: Just to clarify, 1gbps (gigabits per second) is about 125MB/sec, minus overhead. 2gbps would be about 250MB/sec, give or take.
 

This is getting annoying. The deeper I look the more errors I find
 
My fix seems to work. I tried my fix with the current uboot firmware´s supplied by armbian and these are my findings:
 
- 1200 MHz: working fine, sbc-bench: http://ix.io/1BQ9
- 1000 MHz: I´m still trying to find out why this wont work for me
- 800 MHz: working fine, sbc-bench: http://ix.io/1BQr
- 600 MHz: could work fine, if the cpufreq init code would not mess up; at the moment it is just running at 300 MHz, sbc-bench: http://ix.io/1BQF
 
The problem with the 600 MHz is, that the parent clocks are running at 1200 MHz and a divider by 2 is used, but this is a problem for the cpufreq init code, because the cpu core frequency is 600 MHz and the cpu frequency which gets reported to the kernel is 600 MHz divided by the divider found in the dvfs which is 2.
So to fix this problem, the parent frequency has to be taken to report the right frequency to the kernel. As I don´t intend to run at 600 MHz I will ignore this problem, but regard it as reported with this post. If someone wants to fix it and needs more information just ask.
 
When I found out what the problem with 1000 MHz is and I fixed it, I will post the needed patches for fixing the cpu frequency for 800, 1000 and 1200 MHz.

I have an EspressoBINv7, I'm running Armbian 5.72 (Stretch) with kernel 4.19.  It appears to be supported.

I have an EspressoBINv7, I'm running Armbian 5.72 (Stretch) with kernel 4.19.  It appears to be supported.