Peter Lindener

Igor, jernej, valant Thanks for sharing your thoughts. here is dive into manging ARM boot system complexity issues. sleeping on it overnight, I concur with Valant's perspective. Proposed roadmap towards ending up with less long run boot system cruft: 1. our first generic ARM system virgin SD-card boot challenge concerns unifying the various SD-card master boot record offset entry points execute a unified code image. Accomplishing a full unification, such that all SOC's can boot from the very same SD_card image may be near impossible, as suggested in the "UEFI on Top of U-Boot" paper mentioned prior. this may be made even more challenging if different (ARMv7 vs. ARMv8) SOC's, initiate there execution using a different instruction set or execution mode. Neither u-boot or BareBox have attempted to fully resolve the need for unification relating to a multiplicity of partition table formats, boot code image formats, various entry points, ARMv7, ARMv8 and (GPU in the case of raspberry pi) instruction sets. While a partial unification of these various ARM SOC boot paradigms may be possible.. The closest we are may be able to come to unifying the need to handle this multiplicity of board specific SD_card boot images, may involve modifying the SD_card image flashing application such that it would have the capability to merge board specific boot code with an otherwise fully unified, generic, (possibly BtrFS based) ARMbian OS image. While accomplishing this flash image time code merge may be a challenge, taking this approach could help to resolve some of the issues related to the growing number of ARM SOC boot paradigmes. On the other hand, I have yet to find any actual instance of a real "mutual exclusion", that could not be overcome by way of custom crafted master boot record / GPT FAT file system image.. That is: most of these boot formats have some level of flexibility in there layout. the only thing that may be more difficult might be if the same code entry point were used with a pair of mutually exclusive instruction sets.... or am I missing something here? 2. (Optionally) provision the first SD_card partition as a (v?)FAT file system. I propose that this FAT partition would NOT be involved in the initial (BtrFS based boot process) but instead to ease initial configuration and UEFI migration issues.. more on this later. 3. provide BtrFS, boot, Snapshot and Rollback support for (possibly multiple) BtrFS boot images. see: http://git.denx.de/?p=u-boot.git;a=commit;h=b6ee860b87d8accedfb7a1fd5414f7c483603234 and: https://gitlab.labs.nic.cz/search?utf8=✓&search=btrfs&group_id=&project_id=254&search_code=true&repository_ref=master 4. place primary system_config, device_tree vars within this larger BtrFS file partition possibly under /boot/.. to easy legacy migration issues, provide the capability of explicit deference by way of cross file system symbolic linking, such that these files can, if preferred, exist on the (optional, UEFI) FAT partition instead possibly mounted @ /UEFI/.., where they can be more easy modified (possibly by a MS_Win machine) before the SD-card image is booted. for those who are falling in love with UEFI (not me, as I am not happy that UEFI can retain active code beyond boot, and thus provide a home for the lurking of root kits).. the BtrFS based boot could then pass control to a UEFI boot system placed within this FAT partition.. I think this option would be good to have, but I am suggest that our longer term boot system roadmap may want to explore utilizing just BtrFS... prepare for the day we might leave a FAT partition behind. that is: Primary boot control / trust would be placed with the SD_card's BtrFS file system image where system Snapshots and Rollback can also be utilized also with (possibly multiple) boot images. 5. System security: support signing / verification of initial bootstrap image, BtrFS, and any UEFI images. 6. longer run: provide for SD-card BtrFS file system access from MS_windows machines. see: https://github.com/maharmstone/btrfs integrate BtrFS access capability along with a system_config_editor into SD_card image creation programs such as Etcher. Taking this approach makes it very easy remove this legacy FAT file boot system cruft from future systems. Hope this helps ARMbian's longer term boot system roadmap discussion get off to a good start. -Peter

Recently U-Boot has acquired support for UEFI (the Unified Extensible Firmware Interface). how may this effect ARMbian's boot system roadmap? see: https://www.suse.com/docrep/documents/a1f0ledpbe/UEFI on Top of U-Boot.pdf

I think both perspectives have some validity... the amount of user interactivity on first boot likely should be configurable.. all way down to none at all, (build time encrypted root and first user passWords) as perhaps Zador may be suggesting... At the other extreme, ARMbian may someday support the building of generic ARM (v7, v8)? SD_card images, that figures out at run time what it's booting up on.. perhaps by way of UEFI..? not sure. While it's likely the SOC's could be differentiated, maybe not the target board name?.. in this case a runtime board selection menu would likely be good. then a fully generic ARMbian image may still be some a way off in the future.. here I see Zador's perspective that the ARMbian community as a whole may be able to move forward more efficiently if the number of SD_card images in common use can be minimized. There will often be the need for first boot user interaction with regard to additional customization. that was not preconfigured in a first_boot_config file. In summary: a spectrum of ARMbian image configurations. ranging from a fully generic bootable SD-card.. and at the other extreme the possibility that ARMbian may some day support a directory of image board_specific_defualt_kernel_config files for the easy minimization of one's custom made ARMbian image. A well organized framework for first boot interactive config seems in some situations necessary. -Peter

Yes, I get it that when using an Ext4 file system, one can just use a "swap file", such is not the case with BtrFS. of course there is this patch: https://lkml.org/lkml/2014/12/9/718 while I'm not keen on the idea of swap space mounted on the one and only SD-card in a system... one does have to admit, that if one is to create a swap partition on the SD-card, it would be best to place it near the end... BEFORE growing the Root File system.. as you Zador points out, the need for swap space is application specific.. The resize2fs script, code seems to already implement some of the "accept size and percentage input" functionality Zador mentions. but takes it's input from file system customization vars, rather than defaulting to prompt for user input, if they do not exist. I'm posting this messages as I'm interested in using the BtrFS instead of Ext4. Zador has already suggested adjusting the nightly build to default build to use BtrFS.. I find fixed, a full size SD-card image size build to be prohibitively big to handle for even a 8 giaByte SD_card. If the BtrFS advantages of snapshot / rollback is to become part of the ARMbian ecosystem. it would seem we will be needing to adapt to ARMbian BtrFS SD-card image deployment. I agree that building up a user friendly first boot dialog (that could be avoided by way of system customization vars) if well designed and effectively implemented, would take a amount bit of effort. That said: It kind of goes without saying that this interaction is the first experience new uses will have with ARMbian. In summary: It would be great if the requirement for any first boot user interaction could be completely mitigated by way of a custom_install_config file that could be placed in either the current Ext4 based boot partition or the BtrFS rootFS ( not all operating systems may be able to access the BtrFS root FS tree ) If EFI boot is in ARMbian's future the boot partition may eventually want to be FAT?.. The level of interaction provided for custom Kernel configuration during the ARMbian build process. illustrates how nice things can be with regard to ARMbian's ease of use. I'm just asking that newcomer's might also enjoy the same kind of user friendly interaction during there first boot experience... I'll help test, and provide quality new user feedback. I am quite in awe of the build system you folks have constructed. I realize ARMbian developer's often find ther need to carefully choose there priority's. I'm currently doing my best learning how to use Git, let me know how best I can help. -Peter

from your replies: I gather the first boot Ext4 resize delay (still an issue) others have mentioned (likely for a larger, 64Gig SD-card), apparently are not much on an an issue when resizing a BtrFS file system. While at some point most users will in some way connect to the internet, the whole point of providing most of the ARMbian OS as an SD_card image might be able to come up running without the need for an additional downloads of: wget https://mirrors.kernel.org/ubuntu/pool/universe/z/zram-config/zram-config_0.5_all.deb yes, "it's easy" if your ARMbian target on line.. but it may not be during first boot? New ARMbian users (those who have not yet created a customized first boot script) would likely prefer to be asked (with defaults) how the ARMbian SD_card image should be handled.. this includes the possible size of a Linux swap partition near the end, as well as the nature RootFS partition / FS resize operation. That is: I think, blind first boot SD_card state change should be avoided. On target boards that provide additional NV store beyond the SD_card, this first boot user interaction would likely be a good opportunity to prompt for an image install in addition to any options for a possible SD_card resizing operation. Since one is already insisting that users set the root passWord.. it would be nice if new ARMbian users can easily understand that ARMbian SD_card image is being modified... and yes it's great that more advanced users can customize most all of this, then lets keep in mind that it's newcomers who will help grow the ARMbian community yet further.

Tkaiser and Zador - Thanks for giving ARMbian's adjustment towards supporting the BtrFS some additional thought. from my perspective resizing of SD_card partitions / File system images should not be automatic on first_boot, but instead queried if should be done during first_login just after ARMbian's mandatory PassWd adjustment. I make this point, as a long delay during first boot, makes it hard for newbies to know what is going on, not to mention increasing any challenges associated with debugging the initial boot process. I'm not sure what how important it is for Linux to have a swapFile (in my case: NanoPi Neo2 has 512meg of RAM).. sure using ZRAM might help,.. then as Tkaiser points out swapping to/from the SD_card (the only mass storage on a Neo2).. is not a very attractive solution.. since the SD_card resizing logic and this SD-card swap space allocation are both issues that are somewhat unique to systems that run with just the SD-card... it might make sense to also ask (with defaults) how one would like to configure the SD-card/ ZRAM swap space options... Then these additional swap space parameters would be factored into the SD-card's partition adjustment / file system adjustment logic... Yes: taking an initial (post resize?) BtrFS snapshot sound like it would be a good thing! -Peter

Thanks for the assistance getting acquainted with ARMbian's first boot, SD_card resizing logic. see; https://github.com/armbian/build/blob/54d71ef32ff0ce51bafeb26b2a7f864241316e55/packages/bsp/common/etc/init.d/resize2fs I'm still getting acquainted with the logic in this bash script.. a pair of Question: on lines 112 and 127, I find: resize2fs $rootpart >>${Log} 2>&1 At first this looked like some kind of recursive invocation of this very same script. But now after reading the code a second time... It's clear there is a near miss on a same name collision between: this bash script, located in the target at: /etc/init.d/resize2fs and the program this script calls, located in the target at: /sbin/resize2fs I realize that near miss program / script name collisions are an excellent way to confuse new-newcomers like your's truly. In this case, we would seem just a minor adjustment in the program search path away from the fun of an infinite first boot recursive script invocation. Confusing newcomers aside, as we are about to extend this logic to also work with BtrFS root file systems, maybe it would be good to rename this SD_card root file system expansion script, to have a more clarifying name, such as "expand_last_fileSys" or "Expand_root_fileSys" that would not create this kind of same name confusion, as well the potential bugs of a recursive script invocation during first boot? ---------- also: While I'm new to using best GIT practices to submit code changes. I have cleaned up the section of this scripts code that provides for fdisk's control via stdin. see: attached test script... with workaround for bash printf \n truncation bug (bash printf will often omit trailing \n's).. cheers -Peter Test_FDisk_Cntl.sh

Zador- It seems, as I have been reading that Resizing the SD_Card's root file system happens in two stages: 1.resizing the underlying file partitions and then 2. resizing the file system (Ext4 or BtrFS) logic. It's the first part, that seems tricky to do while the Root file system is still mounted... Ahh.. as you point out, I see that the SD_card partition resize logic has now also been folded into: https://github.com/armbian/build/blob/master/packages/bsp/common/etc/init.d/resize2fs has this code settled into its final resting place, of might the partition resize logic end up being re-factored to be a separate script called by the File system specific logic one level above? -Peter

I read through nand-sata-install.... did not find, did you instead intend to reference: armbian-lib/scripts/firstrun see: https://github.com/armbian/build/blob/master/packages/bsp/common/etc/init.d/resize2fs line# 92: # Start resizing... lines#101-106 fdisk ${DEVICE}) line# 108: partprobe ${DEVICE} ... line# 112: (for Ext4 file system resize) then calls: resize2fs for a good explanation of what is going on here with the partition tables see: https://raspberrypi.stackexchange.com/questions/499/how-can-i-resize-my-root-partition to resize a BtrFS file system we would call: btrfs filesystem resize [max, $RootFS_size].. see: https://docs.oracle.com/cd/E37670_01/E37355/html/ol_use_case2_btrfs.html instead of calling resize2fs (for a Ext4 file system resize) http://www.legendiary.at/2013/10/06/resize-btrfs-filesystem/ While making any adjustments to ARMbian's SD_card rootFS resize logic.. it may be good to see the day when uBoot knows how to read/boot directly from a BtrFS root file system.. see: https://github.com/falstaff84/u-boot/blob/master/fs/btrfs/btrfs.c and: https://patchwork.ozlabs.org/patch/300593/ I take it that this capability will at some point also land in the ARMbian OS builder ? I don't mind doing my share of homework and testing.. ..then it's always good to point others towards the file they might actually need to looking at. Please forgive me, I'm still learning how to do this.. -Peter

T-Kaiser - While computing $imagesize and $sdsize will eventually become an issue.. Do we have a way to expand the underlying partition of an SD card built with a BtrFS root file system while it is mounted as / (root)..at this point ?. that is it seems: btrfs filesystem resize max only works after expanding the underlying partition containing the BtrFS root file system Can this partition resize be accomplished while mounted ?.. or does this problem also need solving before we can dream about expanding BtrFS root file systems on first boot..?

I just finished building BtrFS_Armbian_5.32_Nanopineo2_Ubuntu_xenial_dev_4.11.10.img for an 8GigByte SD card.. once I got the resulting image to actually fit on the card with a config-default.conf file containing: ROOTFS_TYPE=btrfs FIXED_IMAGE_SIZE=7200 It seems this fixed image size specification is for just the BtrFS partition size and does not include thee Ext4 boot partition that is also included in the SD card image.. after 4 attempt at building a Xenial system with a BtrFS root volume finally fits within the SD card.. ->df -h -T yields: /dev/mmcblk0p1 ext4 58M 25M 32M 44% /boot /dev/mmcblk0p2 btrfs 7.0G 500M 5.9G 8% / Changes to /etc/network/interfaces seem remain following a reboot... the good news is that ARMbian's support for BtrFS root file systems, with U-boot support via a small Ext4 boot partition seems to be working.. Ya! way to go!... That said... since not everyone ends up using the same size SD card, and burning full card size flash images being a fairly slow process.. I know I'm longing for another ARMbain team miracle here.. but it would be great if we figured out how to do dynamic BtrFS partition expansion, similar to how nice the ARMbian team has already done with Ext4 first boot SD card partition expansion.. see: Resizing a Btrfs File System.. https://docs.oracle.com/cd/E37670_01/E37355/html/ol_use_case2_btrfs.html Then I'm Already a very happy camper... Thanks for the great Linux build system!... the ARMbain team made my day! -Peter

Zador- thanks for the heads up with respect to Docker's Overlay2 over BTRFS file system issues.. but this may not be for long.. see: https://github.com/moby/moby/commit/f64a4ad008e68996afcec3ab34a869887716f944 I'm already running Linux Kernel -4.12.3-041203-lowlatency... But I'm not experienced enough to debug problems on the leading edge of Docker File systems development... so it might be prudent to sit this one out and continue my ARMbian build effort inside VirtualBox.. Thanks for the quality guidance! -Peter

Zador, Thanks for further clarifying, Your reasoning seems well founded. you mention: "Unfortunately VirtualBox storage implementation may significantly affect the performance even with caching." Up this point I've been running the ARMbian build procees inside a Virtual Box VM via SSH. My build machine is a DELL XPS15 laptop is equipped with a 1Terabyte SSD (and 32 GigBytes of RAM). Currently running Linux Mint 18.2 Cinnamon 64-bit (? a Ubuntu 16.04 Xenial derivative ?), using BTRFS file system that also supports Neon KDE dev. instead of running the ARMBian build process inside of VirtualBox.. would you recommend running ARMbian builds inside a Docker package, instead? here : https://github.com/armbian/documentation/commit/95f9f0826b24fce7d4af680a738fcee5d2e3dd73 you wrote: "Using Xenial build host inside containers is **highly recommended**" this seems promising... will using Docker at this point also work when doing compleat SD card image ARMbian builds? Please provide links to latest documentation for how to use a Docker container to do ARMbian builds? Thanks -Peter

Zador- Thanks for your efforts to clarify... Please correct me If I'm still in need of the right clue... 8 GigaByte flash cards are now so prevalent that the extra effort of pruning the size of the SD flash card image to minimal size, (by omitting unused loadable Kernel modules) seems less than productive for most ARMbain builders... (Question: is the following correct?) Source code for all of ARMbian's Kernel modules and drivers will cloned into the local build machine from https://github.com/armbian/build. One can customize the ARMbian Linux kernel's build configuration, doing so is usually unnecessary. doing so will avoid compiling unused driver and module code, and also the related flash image consumption by deselecting that functionality with ARMbians interactive kernel-configuration menu.. Since compiling a module only happens once or when there is a source code.. taking the extra effort to customize the kernel build process, ends up only saving time during the first build.. or when a lot of (unused) kernel code has changed. In summary: ARMbian builders are welcome to create target_board_customized_ARMbian_kernel-configuration files. that will save some compilation time when doing a board specific image builds.. but this optimization of the ARMbian build system seems a bit premature by those currently looking after the ARMbian OS build system. on the other hand, ARMbian kernel builders are still free to do this. Eventually the ARMbian build system may contain a directory that holds board_specific_default_kernal-configuration files., to support just this kind of build system optimization. ..but this can wait... as the ARMbian system image builder already seems the best one out there! -Peter ---> 3 Hours later... after aborting my latest attempt a full image build.... I should have heeded your reflection: "Userspace applications selection depends mostly on the base release (Debian or Ubuntu) and image type (derver or desktop)".... Caution a build of Ubuntu will default to "Desktop" not Server (VERY HEAVY)... that said...it might be good if the default build tended towards the liter side. Zador In reflection: I realize that you may be focused on efficiency when building everything at once. but in all honesty the way things seem to be going the ARMbian team seems to have wrapped a very heavy set of build images around there necks...and soon enough your teams ability to stay afloat as the Linux OS continues to put on weight will make things much more difficult, even for those who have learned to pull off the herculean feat that the ARMBian OS builder currently is. Please don't get me wrong hear.... I VERY MUCH respect what your team has accomplished... even Linus Torvalds is admitting that Linux has grown to heavy... and its clear that anyone who chooses to do Kernel / driver development is wading into the deep end... Clarifying the situation... If it were reasonable to bring up locally compiled GigEMAC drivers for my cluster of NanoPi_Neo2's., without also draping the full weight of the entire Linux build tree around our necks.... I would be working on supporting OpenPowerLink hard real-time networking. Instead, I'm worried if your team working miracles supporting the ARMbian OS build systems, will mange to come up for air the next time. While I understand that your team is focused on ARMbian's production build process.... It might be good to keep in mind that most every other Linux developer ends up focusing there day to day development and debugging work on just a few target platforms at a time.. and without these Linux Kernel hacker's there would not even be a Linux OS for ARMbian to make "easy" to build.. I realize that extending the ARMbian dependency framework such as to lightenn the single platform build load will not be easy... but to just let Linux keep gaining weight... and not have a plan for dealing with the blot.... is sure to fail in the longer run... unfortunately for me that "longer run" has just happened... Even with a brand new DELL XPS quad core i7, the generic build process as things stand is just to heavy.! Maybe all it will take is strategically litening a few of the build defaults... Or perhaps a set of well crafted board_specific_kernel-config files. It would sure be nice using dependency analysis, if one could convince GIT clone to only download the parts of the Linux build tree that will actually be needed for a particular target.. and likewise save on source compilation time... Perhaps your team should exchange notes on how to handle this with Linus... I'm sure he has been thinking about it, I gather he would be greatful if the ARMbian team ended playing a part in the solution.. Please forgive me if I'm sounding less than constructive.. I would like to see the ARMbian team relize it's dreams. all the best -Peter -------

thanks Zador for your reply. If I understand correctly.?.. the current implementation of the ARMbian build system (quite nice as it is) take's the approach of generalizing the Linux image build process to the greatest extent practical. that is: with a default Kernel configuration for given CPU family: variations in Linux Kernel functionality are accommodated by the selection of which loadable Kernel modules to place in the Kernal's RAM at run time.. I now understand that with ARMbian's default-configuration, all kernel modules available for that CPU family will be loaded into the local Git tree clone and each (re)compiled (as needed) during the build process... Question: with ARMbians default kernel configuration... do Kernel modules not required for a given target board also end up included in the board customized flash memory image?... or are they left out during the final fash image build process? Clarifying my original inquiry: ARMbian's build system is already quite nice.. then, my fantasy that board customized ARMbain build VM's will eventually at some point be light weight will have to wait, at least until the dust settles around ARMbian's (rather Herculean) build system development. as a warm up, I've been leaning to build ARMbian Kernel images, before proceeding onward toward to build an entire board specific (NanoPiNeo2) ARMbian system SD card image. additional Question: is there anything else one might want to understand, with regard to the target specific build / (app) selection process, when building ARMbian flash images targeted towards a specific target board within a CPU family?