Minimizing board specific Kernel builds

[Peter Lindener]

new to the ARMbian build process..   seems very well put together.

this is my second attempt at a fresh ARMbian Kernal build in a clean VBox Mini_Ubuntu 16.04 VM.

this Kernel build  (successful) was invoked by:

 

git clone --depth 1 https://github.com/armbian/build

cd build

./compile.sh EXPERT=yes BRANCH=dev RELEASE=xenial BOARD=nanopineo2 KERNEL_ONLY=yes

 

I am still easing into the build process,  at the interactive prompt,  I opted not to change the kernel's configuration.

 

During the build, I noticed that many modules unrelated to my target NanoPiNeo2 board's hardware..

.. ( yes I know it's still a WIP,  then for me it's likewise. )...

 

My question: is it's possible to restrict the ARMbian Git download and Kernel Build process to fetch and compile only the modules/code

required for a specific targeted board(?s).?...      

Is this an issue with the (board specific?) default kernel-configuration still needing some additional work..

or am I just missing something in my command line specification of the build process?

 

 

 

 

 

 

[zador.blood.stained]

10 hours ago, Peter Lindener said:

My question: is it's possible to restrict the ARMbian Git download and Kernel Build process to fetch and compile only the modules/code

required for a specific targeted board(?s).?...      

No, unless you edit the kernel config by yourself and use the edited config for the build process, and it will affect only the compilation, local sources copy will still contain everything.

 

10 hours ago, Peter Lindener said:

Is this an issue with the (board specific?) default kernel-configuration still needing some additional work..

or am I just missing something in my command line specification of the build process?

There is no "board specific" kernel configurations. and it's not an "issue" - one kernel configurtation is shared between multiple boards (i.e. all H5 and A64 boards in case of Neo2). You can always minimize the config for yourself, but default ones will include as much potentially useful features as possible.

[Peter Lindener]

thanks Zador for your reply.

   If I understand correctly.?..

   the current implementation of the ARMbian build system (quite nice as it is)

take's the approach of generalizing the Linux image build process to the greatest extent practical.

   that is: with a default Kernel configuration for  given CPU family:  variations in Linux Kernel functionality

are accommodated by the selection of which loadable Kernel modules to place in the Kernal's RAM at run time..

 

   I now understand that with ARMbian's default-configuration, all kernel modules available for that CPU family will be loaded into the local Git tree clone and each (re)compiled (as needed) during the build process...

 

   Question: with ARMbians default kernel configuration...   do Kernel modules not required for a given target board also end up included in the board customized flash memory image?...   or are they left out during the final fash image build process?

 

   Clarifying my original inquiry: ARMbian's build system is already quite nice..   then, my fantasy that board customized ARMbain build VM's will eventually at some point be light weight will have to wait, at least until the dust settles around ARMbian's (rather Herculean) build system development.

 

   as a warm up, I've been leaning to build ARMbian Kernel images, before proceeding onward toward to build an entire board specific (NanoPiNeo2) ARMbian system SD card image. 

 

   additional Question: is there anything else one might want to understand, with regard to the target specific build / (app) selection process, when building ARMbian flash images targeted towards a specific target board within a CPU family?

 

    

 

   

 

    

  

 

 

 

[zador.blood.stained]

12 hours ago, Peter Lindener said:

with a default Kernel configuration for  given CPU family:  variations in Linux Kernel functionality

are accommodated by the selection of which loadable Kernel modules to place in the Kernal's RAM at run time..

If we are talking about more or less modern kernels then modules are not loaded unless they are required by internal or external hardware, loaded explicitly by user or software. While some features may increase the size of the base kernel image, it has to be done.

 

12 hours ago, Peter Lindener said:

Question: with ARMbians default kernel configuration...   do Kernel modules not required for a given target board also end up included in the board customized flash memory image?...   or are they left out during the final fash image build process?

All compiled modules are packaged into the kernel package. This is not something unique since Debian and Ubuntu kernel packages are made in a similar way. And given that currently it's hard to fine and SD or eMMC device of less than 8GiB I can't say that disk space is taken into consideration. It's not LEDE/OpenWRT which targets devices with as low as 8-16 MiB of internal storage for everything - kernel, compressed rootfs, compressed r/w data storage.

 

12 hours ago, Peter Lindener said:

Question: is there anything else one might want to understand, with regard to the target specific build / (app) selection process, when building ARMbian flash images targeted towards a specific target board within a CPU family?

Userspace applications selection depends mostly on the base release (Debian or Ubuntu) and image type (derver or desktop), there are no board specific tweaks (with minor exceptions) by default. While it may not be optimal because the resulting image may contain packages that won't be used on the target hardware, it results in a more effective caching. So defaults are optimized towards mass production of images since rebuilding all supported targets takes quite some time even on a server-grade hardware, unless multiple build processes run in parallel (which has some limitations too).

[Peter Lindener]

Zador-

     Thanks for your efforts to clarify...   Please correct me If I'm still in need of the right clue...

     8 GigaByte flash cards are now so prevalent that the extra effort of pruning the size of the SD flash card image to minimal size, (by omitting unused loadable Kernel modules) seems less than productive for most ARMbain builders...    

(Question: is the following correct?)

Source code for all of ARMbian's Kernel modules and drivers will cloned into the local build machine from https://github.com/armbian/build. 

One can customize the ARMbian Linux kernel's build configuration,  doing so is usually unnecessary.

    doing so will avoid compiling unused driver and module code, and also the related flash image consumption by deselecting that functionality with ARMbians interactive kernel-configuration menu..    Since compiling a module only happens once or when there is a source code.. taking the extra effort to customize the kernel build process, ends up only saving time during the first build.. or when a lot of (unused) kernel code has changed. 

    In summary:  ARMbian builders are welcome to create target_board_customized_ARMbian_kernel-configuration files. that will save some compilation time when doing a board specific image builds..   but this optimization of the ARMbian build system seems a bit premature by those currently looking after the ARMbian OS build system.  on the other hand, ARMbian kernel builders are still free to do this. 

 

    Eventually the ARMbian build system may contain a directory that holds board_specific_default_kernal-configuration files., to support just this kind of build system optimization. 

    ..but this can wait... as the ARMbian system image builder already seems the best one out there!

    -Peter  

 

---> 3 Hours later... after aborting my latest attempt a full image build....

    I should have heeded your reflection: "Userspace applications selection depends mostly on the base release (Debian or Ubuntu) and image type (derver or desktop)"....  

Caution a build of Ubuntu will default to "Desktop" not Server (VERY HEAVY)...

that said...it might be good if the default build tended towards the liter side.

 

 Zador  In reflection: I realize that you may be focused on efficiency when building everything at once. 

but in all honesty the way things seem to be going the ARMbian team seems to have wrapped a very heavy set of build images around there necks...and soon enough your teams ability to stay afloat as the Linux OS continues to put on weight will make things much more difficult, even for those who have learned to pull off the herculean feat that the ARMBian OS builder currently is.

    Please don't get me wrong hear.... I VERY MUCH respect what your team has accomplished...

even Linus Torvalds is admitting that Linux has grown to heavy...   and its clear that anyone who chooses to do Kernel / driver development is wading into the deep end...

   Clarifying the situation...    If it were reasonable to bring up locally compiled GigEMAC drivers for my

cluster of NanoPi_Neo2's., without also draping the full weight of the entire Linux build tree around our necks....   I would be working on supporting OpenPowerLink hard real-time networking.

   Instead, I'm worried if your team working miracles supporting the ARMbian OS build systems, will mange to come up for air the next time.    While I understand that your team is focused on ARMbian's production build process.... It might be good to keep in mind that most every other Linux developer ends up focusing there day to day development and debugging work on just a few target platforms at a time..    and without these Linux Kernel hacker's there would not even be a Linux OS for ARMbian to make "easy" to build..

   I realize that extending the ARMbian dependency framework such as to lightenn the single platform build load will not be easy...   but to just let Linux keep gaining weight... and not have a plan for dealing with the blot.... is sure to fail in the longer run...   unfortunately for me that "longer run" has just happened...   Even with a brand new DELL XPS quad core i7,  

   the generic build process as things stand is just to heavy.! 

Maybe all it will take is strategically litening a few of the build defaults...  Or perhaps a set of well crafted board_specific_kernel-config files.

    It would sure be nice using dependency analysis, if one could convince GIT clone to only download the parts of the Linux build tree that will actually be needed for a particular target..   and likewise save on source compilation time...   Perhaps your team should exchange notes on how to handle this with Linus...    I'm sure he has been thinking about it, I gather he would be greatful if the ARMbian team ended playing a part in the solution..    Please forgive me if I'm sounding less than constructive..

I would like to see the ARMbian team relize it's dreams.

 

    all the best

      -Peter

 

-------  

       

  

     

         

 

     

 

 

     

 

     

    

 

[zador.blood.stained]

9 hours ago, Peter Lindener said:

   8 GigaByte flash cards are now so prevalent that the extra effort of pruning the size of the SD flash card image to minimal size, (by omitting unused loadable Kernel modules) seems less than productive for most ARMbain builders...    

(Question: is the following correct?)

Kernel package may be ~15MiB in size packed and ~50MiB unpacked (maybe ~30-50% more in case of arm64). Personally I don't see any reasons for maintaining 70 130 or more separate per board kernel configurations instead of current ~25 (not counting unsupported development ones). And please, what modules do you call "unused"? You would be surprised what ideas may come to people's minds, so enabling all available netfilter options, multimedia support including DVB-T USB tuners and USB web cameras and different I2C/SPI sensors is not a waste - somebody may come and ask to enable this stuff.

 

9 hours ago, Peter Lindener said:

Source code for all of ARMbian's Kernel modules and drivers will cloned into the local build machine from https://github.com/armbian/build. 

 

There is no such thing as "Armbian's kernel". There are separate kernel sources (35 currently), mostly third party (released by the vendor, community or pure mainline from kernel.org). Only one source at the time will be cloned - one needed for the device and branch you selected.

 

9 hours ago, Peter Lindener said:

One can customize the ARMbian Linux kernel's build configuration,  doing so is usually unnecessary.

Even current 35 configs (of different kernel versions so they can't be easily compared side by side) are not easy to maintain by a relatively small team, so people may have to compile a custom kernel to add features that they want (before reqested changes hit the next stable release).

 

9 hours ago, Peter Lindener said:

    In summary:  ARMbian builders are welcome to create target_board_customized_ARMbian_kernel-configuration files. that will save some compilation time when doing a board specific image builds..   but this optimization of the ARMbian build system seems a bit premature by those currently looking after the ARMbian OS build system.  on the other hand, ARMbian kernel builders are still free to do this. 

Again, currently there is no implementation for board specific kernel configs unless yo manually replace the common configuration on each run, but it's relatively easy to provide a customized (i.e. minimized) configuration in place of one of the default ones.

 

9 hours ago, Peter Lindener said:

Caution a build of Ubuntu will default to "Desktop" not Server (VERY HEAVY)...

This is not true, there is no default value for the image type with the exceptions of boards that don't have a display output (and then the default is obviously CLI/server).

And I wouldn't call 1-1.5GiB image (uncompressed) "very heavy", it's not that much for a full OS image. And since the default package selection is targeted towards what you can call a "Raspberry Pi target audience" (where people expect stuff to work out of the box and are confused about the concept of installing/removing packages to change the available functionality) there may be packages that can be safely disabled but probably won't cut the image size below the 1GiB.

 

9 hours ago, Peter Lindener said:

In reflection: I realize that you may be focused on efficiency when building everything at once. 

but in all honesty the way things seem to be going the ARMbian team seems to have wrapped a very heavy set of build images around there necks...and soon enough your teams ability to stay afloat as the Linux OS continues to put on weight will make things much more difficult, even for those who have learned to pull off the herculean feat that the ARMBian OS builder currently is.

And I still stand on my point about the efficiency. People reported that they needed 6 or more hours for the initial image build run but it didn't stop them, and caching will cut most of this time on subsequent runs with the same build parameters.

Build process is usually bottlenecked by the storage (usually if it is a HDD) and then by the CPU power. Unfortunately VirtualBox storage implementation may significantly affect the performance even with caching.

Ccache allows for recompilation time under 5 minutes even with a 10 years old CPU, but implement board specific configs - bye-bye ccache, welcome ~30-50 minutes compilation time for each board on the hardware I have.

Rootfs caching allows rebuilding OS images within 10-15 minutes (including kernel recompilation time, with the same hardware as above), add board specific package selection and you will have 1-1.5 hours build time per image.

 

9 hours ago, Peter Lindener said:

It would sure be nice using dependency analysis, if one could convince GIT clone to only download the parts of the Linux build tree that will actually be needed for a particular target..   and likewise save on source compilation time...  

It's unrealistic to expect something like this implemented ever. As a person who a little bit understands the kernel building process and how configuration and additional parameters like the target architecture interact with the Kconfig architecture I can say that this won't happen. And as a person who a little bit understands the base concept of the Git I can only confirm that implementing the ability to fetch a partial set of objects tied to a specific commit is not something that can be easily done.

[Peter Lindener]

Zador, Thanks for further clarifying,   Your reasoning seems well founded.

 

you mention: "Unfortunately VirtualBox storage implementation may significantly affect the performance even with caching."

 

  Up this point I've been running the ARMbian build procees inside a Virtual Box VM via SSH.

My build machine is a DELL XPS15 laptop is equipped with a 1Terabyte SSD (and 32 GigBytes of RAM). 

Currently running Linux Mint 18.2 Cinnamon 64-bit (? a Ubuntu 16.04 Xenial derivative ?),  

using BTRFS file system that also supports Neon KDE dev. 

   instead of running the ARMBian build process inside of VirtualBox..

 would you recommend running ARMbian builds inside a Docker package, instead?

 

here : https://github.com/armbian/documentation/commit/95f9f0826b24fce7d4af680a738fcee5d2e3dd73

you wrote: "Using Xenial build host inside containers is **highly recommended**"  this seems promising...

will using Docker at this point also work when doing compleat SD card image ARMbian builds?  

    Please provide links to latest documentation for how to use a Docker container to do ARMbian builds?

 

    Thanks

      -Peter

   

 

 

[zador.blood.stained]

56 minutes ago, Peter Lindener said:

   instead of running the ARMBian build process inside of VirtualBox..

 would you recommend running ARMbian builds inside a Docker package, instead?

In general - yes, but

56 minutes ago, Peter Lindener said:

using BTRFS file system that also supports Neon KDE dev. 

I'm not sure if it will work: https://docs.docker.com/engine/userguide/storagedriver/selectadriver/#supported-backing-filesystems

AFAIK Docker on Debian/Ubuntu (and its derivatives) uses aufs as the default storage driver, and it (and the fallback overlay2 driver) requires either ext4 or xfs. Btrfs is supported only when using the btrfs storage driver and documentation says that it is supported only on Docker EE.

 

[Peter Lindener]

Zador-

    thanks for the heads up with respect to Docker's Overlay2 over BTRFS file system issues..

but this may not be for long..

see: https://github.com/moby/moby/commit/f64a4ad008e68996afcec3ab34a869887716f944

   I'm already running Linux Kernel -4.12.3-041203-lowlatency...  But I'm not experienced enough to debug problems

on the leading edge of Docker File systems development...    so it might be prudent to sit this one out

and continue my ARMbian build effort inside VirtualBox..

 

    Thanks for the quality guidance!

        -Peter