Armbian-NG, armbian's little brother project

[AndrewDB]

I would like to announce that I am getting started on an armbian-derived project, Armbian-NG. This is not a fork of Armbian, nor a rewrite, it's a derived project that piggyback's on the excellent work by Igor and the entire armbian development team. I think of Armbian-NG as armbian's little clumsy brother, who is just learning to walk but has great potential. 

In as few words as possible, Armbian-NG will be a Python (set of) script(s) that runs natively on Aarch64 (64-bit ARM) hardware (running Ubuntu 18.04 or later) to build Armbian images. More details  can be found here: https://github.com/AndrewBCN/Armbian-NG

 

As with armbian, anybody is welcome to contribute with code and ideas to Armbian-NG. And since no project is worth its storage space without a logo, Armbian-NG already has one. 

[lanefu]

On 3/22/2019 at 10:04 AM, AndrewDB said:

In as few words as possible, Armbian-NG will be a Python (set of) script(s) that runs natively on Aarch64 (64-bit ARM) hardware (running Ubuntu 18.04 or later) to build Armbian images. More details (but no code yet) can be found here: https://github.com/AndrewBCN/Armbian-NG

So I'm trying to be enthusiastic about this for you, but it's kind of a disappointment to post an announcement like this on the Armbian forum and literally not have any code.   

[chwe]

moved, cause it doesn't fit really into announcements.. For the native builds.. I think the idea is nice but your build-time may be long.. E.g. I did some tests on a RPi2 (e.g. to see if zram helps for kernel-compilation --> in the end compilation seems to be limited by CPU for the RPi2 not ram.. Takes something like two hours by using native compilers..

[lanefu]

Might be fun with distcc ...is that still a thing?    I guess that would defeat the simplicity intensions 

 

There's always EC2 A1 instances....

[NicoD]

On 3/22/2019 at 3:04 PM, AndrewDB said:

In as few words as possible, Armbian-NG will be a Python (set of) script(s) that runs natively on Aarch64 (64-bit ARM) hardware (running Ubuntu 18.04 or later) to build Armbian images.

 

46 minutes ago, chwe said:

. I did some tests on a RPi2

That's not going to run on the RPi2 You'll need to search another use for it
Would be nice for the faster ones like the RK3399's and Odroid N2. But it seems to me a lot of work that needs to be done. Best of luck @AndrewDB

[AndrewDB]

On 3/24/2019 at 3:11 AM, lanefu said:

So I'm trying to be enthusiastic about this for you, but it's kind of a disappointment to post an announcement like this on the Armbian forum and literally not have any code.   

Sorry, but before I start typing I like to think for a long time about the general skeleton of the program and the data structures I am going to use.

The Python code is WIP, as I am still deciding on the Python libraries I am going to use. I'll push something to the repository tonight,  but I don't expect to have any functionality before a week or so, and I am guessing it will take around a month before Armbian-NG can generate an image that can be booted.

[AndrewDB]

14 hours ago, chwe said:

moved, cause it doesn't fit really into announcements.. For the native builds.. I think the idea is nice but your build-time may be long.. E.g. I did some tests on a RPi2 (e.g. to see if zram helps for kernel-compilation --> in the end compilation seems to be limited by CPU for the RPi2 not ram.. Takes something like two hours by using native compilers..

I have been compiling the Linux kernel natively on some Amlogic TV boxes I have running Armbian. I can compile the Linux kernel in 44 minutes on an S912 box. Using distcc the time goes down to around 15 minutes. The distcc build farm is basically made up of Amlogic TV boxes connected with a 100mbps switch.

I have posted a HOWTO about distcc on aarch64 here: http://wiki.loverpi.com/specs:sbc:distcc-kernel-compile

 

It takes around two hours to build an image using Armbian on my Thinkpad-T420 with a relatively fast Internet connection, start-to-finish.

I expect Armbian-NG will take less than an hour to build an image using distcc on my low-cost build farm.

 

[AndrewDB]

13 hours ago, NicoD said:

 

That's not going to run on the RPi2 You'll need to search another use for it
Would be nice for the faster ones like the RK3399's and Odroid N2. But it seems to me a lot of work that needs to be done. Best of luck @AndrewDB

Thanks NicoD! 

[AndrewDB]

14 hours ago, lanefu said:

Might be fun with distcc ...is that still a thing?    I guess that would defeat the simplicity intensions 

 

There's always EC2 A1 instances....

Using distcc is a good way to save time during Linux kernel compilations, in my experience, and it's quite simple to setup on a bunch of SBC's or TV boxes.

Besides saving time, it also saves energy: my build farm has a total power consumption of around 25W, vs around 65W for the Intel processor in my laptop. 

[AndrewDB]

15 hours ago, NicoD said:

That's not going to run on the RPi2

Indeed Armbian-NG will only run on Aarch64 (64-bit ARM) (it says so in the README). I just wrote the code that checks for that:

import os

...

# Check the underlying architecture, must be Aarch64
    arch = os.uname()
    print(arch)
    if not arch.machine == "aarch64":
        print('Armbian-NG must be built on ARM 64-bit machine (Aarch64)')

 

[chwe]

8 hours ago, AndrewDB said:

25W, vs around 65W for the Intel processor in my laptop.

hmm measuring consumption on a notebook is hard (only idea I have would be disconnect the battery and measure from PSU)..

 

8 hours ago, AndrewDB said:

It takes around two hours to build an image using Armbian on my Thinkpad-T420 with a relatively fast Internet connection, start-to-finish.

I expect Armbian-NG will take less than an hour to build an image using distcc on my low-cost build farm.

I don't criticize the idea.. Especially building on native environments does IMO make sense. Let's see what comes out from. Bookmarked your repo cause I've some bash scripts which I may also bring to python in the future, so it's always good to see how such stuff can be solved (I would describe my python skills somewhere near to non existing, and my bash scripts skills somehow functional ).

 

If you cash stuff e.g. builded rootfs and reuse it you will only need to compile u-boots and kernels which will save you some time. Interested how your project is going.

[AndrewDB]

6 minutes ago, chwe said:

...

 

If you cash stuff e.g. builded rootfs and reuse it you will only need to compile u-boots and kernels which will save you some time. Interested how your project is going.

 

I'll keep posting the progress here.

Yes there are many ways to save a lot of time from one image generation to the next, by reusing files that are already downloaded or compiled.

I am also thinking about using a directory structure and configuration files identical to those of Armbian, so that people who are familiar with building Armbian images would be able to just switch over to Armbian-NG quite easily.

I also am having quite a bit of fun programming in Python again, after almost a decade of not writing a single line of Python. 

[Tido]

On 3/25/2019 at 12:46 PM, AndrewDB said:

the general skeleton of the program and the data structures I am going to use.

will you go minimal like: https://www.ubuntu.com/core  ?

 

 

@chwe, Coding in Python 01 - Introduction and Getting Started  https://youtu.be/IFk1kfM3Ovc

 

[lanefu]

re: python videos...

these have incredible production quality

 

https://www.youtube.com/playlist?list=PLi01XoE8jYohWFPpC17Z-wWhPOSuh8Er-

 

[AndrewDB]

16 hours ago, Tido said:

will you go minimal like: https://www.ubuntu.com/core  ?

 

 

Hi Tido,

The initial development plan is:

Step 1 - Reproduce exactly the functionality of the Armbian build script (minus support for 32-bit SoC's).

Step 2 - Add distcc support (relatively easy).

Step 3 - Add support for Debian 9.6, 9.7, 9.8 and Ubuntu 18.10 and 19.04 build hosts (also relatively easy).

Step 4 - Add Ubuntu 19.04 build target for S9XXX tv boxes (because that's what I have). An Ubuntu core build target should be relatively easy to add based on that.

Step 5 - Add Alpine Linux build target for S9XXX tv boxes. These are very small images, check the Alpine Linux website for details.

 

Support for other host distributions and build targets will depend on other people's contributions to Armbian-NG. Armbian-NG can potentially support any Linux distribution that is built and has public repositories for Aarch64: Fedora, openSUSE, Arch, Gentoo, etc

 

Edited March 29, 2019 by AndrewDB
More information added

[AndrewDB]

Just a quick update on my progress, as of April 2 2019:

• I just pushed version 0.04 to GitHub.
• 230 lines of code and 70 lines of documentation have been committed in the last 10 days. The Python code is divided in two files: build.py which is really the main program, and ngsupportfunc.py which contains all the support functions as well as more complex functions broken down from the image build process.
• Armbian-NG already actually does something now: it git clones a selectable branch of armbian-build.
• I now have a clearer idea of how to write what Armbian-NG is supposed to do in Python, what libraries to use, etc.
• Yes, Armbian-NG will support optionally using distcc to compile the Linux kernel over a build farm. I know exactly how to program that, and it's actually quite simple. See this HOWTO for details: http://wiki.loverpi.com/specs:sbc:distcc-kernel-compile
• I have begun thinking about how to use the Python logging library https://docs.python.org/3/library/logging.html to implement proper logging in Armbian-NG.
• Steps 1, 2, 3 and 4 above are relatively simple to implement.
• Still thinking about how to implement Alpine Linux image build support.
• I have just added some documentation about armbian / Armbian-NG configuration files.

[Tido]

are you using https://en.wikipedia.org/wiki/Travis_CI   on Github?

[AndrewDB]

14 hours ago, Tido said:

are you using https://en.wikipedia.org/wiki/Travis_CI   on Github?

Hi Tido.

I am not using Travis, and I am not sure it's really needed for a relatively small Python project such as Armbian-NG.

[Tido]

it suits every size I think.

And  what if your idea gets big?

 

// sent from mobile phone //

[TonyMac32]

As someone with *looks around at piles of SBC's* ...   well ok, a problem, I like the idea with distcc.  A little sad my XU4's can't join the party though, especially since you can get clusters with that SoC

[AndrewDB]

Just a weekly update on the progress on Armbian-NG, as of April 10 2019:

• I pushed version 0.05 yesterday.
• The public repository now holds a total of 482 lines of Python code and 102 lines of documentation (in markdown format).
• Almost all the new code is the TUI (terminal user interface), which tries to copycat as much as possible Armbian's ncurses-based dialogues using a Python library called npyscreen. This took me longer to program than I expected because I was held back by a bug "undocumented feature" in npyscreen...  I estimate the TUI code is 80% completed now.
• I also wrote some "internal documentation" (notes) for my personal reference, which I have not uploaded to the repository. But if anybody wants to contribute, this is available, just PM me.
• After the past week's effort into programming the TUI for Armbian-NG, I have renewed respect and appreciation for the amazing programming work done by Igor in writing Armbian. It truly is a wonder! If I am able to do 10% of what Igor did I will be proud already!
• Over the next week I intend to let the TUI code settle down a little bit and work mainly on the kernel downloading and compilation steps.

 

[Tido]

On 4/10/2019 at 12:43 PM, AndrewDB said:

Over the next week

I am curious, how is it going?

 

[balbes150]

Um, I missed a topic that I'm very interested in. My dream is transfer all my assemblies completely on the Assembly for the ARM hardware that would completely get rid of crosscompilation on x86.

[pask]

This is an excellent idea: I have an odroid n2 always on the Internet using a public IP, I'd like to use to compile Armbian for other boards (at the moment a NanoPiM4-v2). Sadly, it seems that the author abandoned, or could not continue this interesting project.

 

On the other side, for what I have understood reading some of the Armbian code, It's not easy to archive the goal of compiling directly on arm64 hardware.

[sfx2000]

8 hours ago, pask said:

On the other side, for what I have understood reading some of the Armbian code, It's not easy to achieve the goal of compiling directly on arm64 hardware.

 

Can't speak for armbian - but there's no difference performance wise on target code between native vs. crossbuilds on GCC - at least not that I've seen.

 

Since the toolchains are already sorted out for Armbian - is it worth the effort to do a native toolchain? Might be, but someone would have to do that effort.

 

Openwrt does build native, BTW, but that's because someone did do the effort to make it happen - but again, I've not seen any difference in performance - I'd rather have a stable AMD64 that can build for any supported ARCH, and that problem is already solved.

[pask]

2 hours ago, sfx2000 said:

 

Can't speak for armbian - but there's no difference performance wise on target code between native vs. crossbuilds on GCC - at least not that I've seen.

 

Since the toolchains are already sorted out for Armbian - is it worth the effort to do a native toolchain? Might be, but someone would have to do that effort.

 

Openwrt does build native, BTW, but that's because someone did do the effort to make it happen - but again, I've not seen any difference in performance - I'd rather have a stable AMD64 that can build for any supported ARCH, and that problem is already solved.

@sfx2000

It's not a question of performance, but of convenience. At least to me.

Small fanless ARM personal computer devices are getting faster and faster on one hand. On the other, they can be easy installed at home or other places where space, energy costs, and even cooling, are an issue. And left there running 24h.

So, you can launch heavy duty compiling tasks and forget them, till the next morning or the evening when you return back home.

 

BTW, I understand that building a working arm64 toolchain could be not easy. Actually, I can't imagine the effort required. I would have to deepen armbian code or be given some hint where to start by somebody more experienced than me.

[Igor]

Modern kernel code - when you use armbian-config way of installing kernel sources (on Debian Buster or Ubuntu Bionic based Armbian) - you only need to go to the source directory and run "make zImage|Image" and you are done. In some case some wireless drivers might have issues with compilers ... but don't take this feature as granted yet.

[balbes150]

Please contact the administration to move this topic to the TV box section. I would like to (as far as possible) continue this direction (native Armbian build directly on ARM devices). For skeptics. Recently, I completely moved the LivreELEC build to the ARM platform (i.e. the entire process of creating LE images and Addons now takes place on ARM rk3399\s922x devices). My dream is (perhaps) to try to transfer the ArmbianTV build completely to the ARM platform and get rid of the x86 dependency.

[sfx2000]

On 4/28/2020 at 8:38 AM, balbes150 said:

My dream is (perhaps) to try to transfer the ArmbianTV build completely to the ARM platform and get rid of the x86 dependency.

 

In personal experience - it's generally doable... and it's really the makefiles for externals like device drivers that are the devils to be solved.

 

In any event - cross-builds produce the same performance as native - ARM on X86, MIPS on ARM, ARM on ARM...

 

For most folks - LEDE/OpenWRT has solved that particular problem...

[balbes150]

On 6/21/2020 at 3:36 AM, sfx2000 said:

In personal experience - it's generally doable...

The issue has already been resolved and the Armbian build works on ARM devices.