[RFC 001] Changes for boards and features implementing

[Igor]

We all know there are several shortcomings which causes mess in the config files and prevent simple implementing of more complex scripting. In order to make build system future proof and to cleanup the exception mess, which is virtually everywhere, I decided to start working on a part of the build system. Now the concept works and it is not that far to be mad if idea is bad

 

packages/extras was moved into this, then board support package and (for now) Cubietruck and Tinkerboard hacks from config/sources/ All others have to be implement into packages and their scripts. It's one time job and it will be much easier in the future, with new boards or functions.

 

New board support packages are now broken into unlimited number of packages. Currently there are three main groups and already present logical packages. Most of present are tested and are fully operational. Mostly its copy/past with bug fixed here and there. Perhaps some bugs were made in this process, but in essence system works - for those two boards. Upgrade path is not determined yet - I only focused on packaging and installing. All those packages can be installed from freshly build ones or from the repository. Each package can have their own number and package is rebuild only if number upstream doesn't exists.

 

This RFC includes preliminary merge of @balbes150 TV boxes fork so it's ATM a bit messy. Cubietruck and Tinkerboard images were tested (Bluetooth briefly, audio need to check again), the rest is not prepared and it requires some manual work. I hope someone else, not just the usual suspects, will help doing this.

 

I will slowly move forward and keep it mergeable/synced with upstream.

 

This approach is more or less only a working proposal for changes. IMHO it's better than what we have now but not perfect.

(WIP) Readme with some more details https://github.com/armbian/build/tree/tvboxes/config/packages

 

You can try this by adding LIB_TAG="tvboxes"

 

Edit:

 

Skeleton is done, images can be build from this branch ... Most but not all of our current families and board specialities has been moved under this packaging system. I tested Cubietruck, RockPro, Tinkerboard. At least they works fine - checked for Bluetooth (TB/CB3) and video acceleration on Cubietruck.

TBD: adjust changes in armbian-config regarding desktop (nodm is gone, lightdm only with possible autologin) and kernel switching.

@JMCC Welcome to try adding perhaps Tinkerboard MM script as a new multimedia/something package. What shall be its name? If some 3rd party packages needs to be placed to the repository, put them here: https://github.com/armbian/upload

 

@gprovost mvebu family is not there yet, but if you got time it should be clear what to do? If not, I wrote lousy docs or scripting is more complicated as before  

 

@selfbg Added Olimex SOM204. Check when possible.

@zador.blood.stained Are packages relations alright? I hope they are future proof. Upgrade is planned from armbian-config and is actually optional. Older packages, except boot and kernel are no more.
 

@tkaiser cpufreq is not overwriting anymore, serial consoles are board property SERIALCON="ttyS0,ttyGS0", board config is reloaded right before packages building. Shell we move hardware-optimisations.sh to per board script? Changes should not affect OMV.

 

@Staars Z28Pro merged but untested

 

@lanefu @5kft @TonyMac32 .... check.

@balbes150 have a lot of work to move his scripting into the build engine. Helping him, testing, fixing, ... @all Check and join if you can.

 

beta.armbian.com can be switched to this new world order ASAP.

Expected merge due: 2/2019 ?

Happy 19!

Edit by lanefu 7/2019:

A project #1 has been created on github.   There are several tasks, which will be tracked as issues on the board.

Edited July 4, 2019 by lanefu
link to project and RFC ID#

[Tido]

From a newbie or outsider perspective, have you have drawn a sketch by hand or something digital how the relation between the scripts look like?

 

[Igor]

2 hours ago, Tido said:

From a newbie or outsider perspective, have you have drawn a sketch by hand or something digital how the relation between the scripts look like?

 

13 hours ago, Igor said:

Upgrade path is not determined yet - I only focused on packaging and installing.

Those relations needs to be visualized and rethink before merge.

[Tido]

14 hours ago, Igor said:

Now the concept works and it is not that far

That said, I had the impression you have already left planning and that your are already doing it ?

Seeing a sketch of your not finished idea would help raise ideas

[Igor]

33 minutes ago, Tido said:

Seeing a sketch of your not finished idea would help raise ideas

The main idea is written here: https://github.com/armbian/build/tree/tvboxes/config/packages It's actually simple. Upgrade might be more problematic.

[balbes150]

@Igor

I looked at the option to neutralize the problem with installing lightdm\nodem packages. I think this may add to the problems. There are ready-made rootfs, which is previously formed and tested. It is used to create a working version of the system, if you try to update the installed packages from it to the current version in the network repositories, you can get to the same problem that we are trying to solve - unsynchronization of packages in an even larger form (since the whole system is updated). The main problem of network repositories is that you can get to a state when some of the packages in it have changed and temporarily lost connections (dependencies). I regularly get on such problems. So I suggest forming all the ROOTFS in one go. All packages that are used from online repositories should be obtained in one transaction (when forming  rootfs cast), if lucky and it turned out to be working, then it can be safely used for further builds, until the next (possibly manual) instructions to form a new version of the archive with ROOTFS (if you manually delete the archive with ROOTFS).

 

 

[Igor]

4 minutes ago, balbes150 said:

It is used to create a working version of the system, if you try to update the installed packages from it to the current version in the network repositories, you can get to the same problem that we are trying to solve

I don't recall to have such problems (only bugs which we made in desktop/bsp dependencies) and I am usually working with a few days old rootfs cache. I understand the problem, but on the other hand ... it must work. Basic packages should not cause such troubles. When user runs the system update, he hit that problem ... if exists. Right? So we better fix this somehow. We don't need to freeze all packages as they are.

 

7 minutes ago, balbes150 said:

So I suggest forming all the ROOTFS in one go.

This can be done, but we will have more cached files. (this and that) desktop with this and that display manager. I am trying to avoid this situation. If possible. Let's leave this problem open. I will try another approach.

 

Another issue is u-boot. We will drop u-boot package installation for all boards. Only compiling, packing and writing. But we need to have "write_uboot_platform" definitions for nand-sata-install and/or manual install/update via armbian-config ...
 

In this case variable ADD_UBOOT is deprecated.

[balbes150]

14 hours ago, Igor said:

I don't recall to have such problems (only bugs which we made in desktop/bsp dependencies) and I am usually working with a few days old rootfs cache. I understand the problem, but on the other hand ... it must work. Basic packages should not cause such troubles. When user runs the system update, he hit that problem ... if exists. Right? So we better fix this somehow. We don't need to freeze all packages as they are.

Yes, this also happens to users (and quite often). When they run the update and as a result, some of the features they used are broken (you have to remove conflicting packages). They have to either write a bug report and wait for a fix, or roll back to previous, compatible versions of packages. When you manually update the system writes, if there are conflicts and the user decides what functions are important to him and how to do (what to remove and what to leave). In an automatic build, you cannot provide all the behaviors and always use the default.

It is possible you that the online repositories used in the build are usually in a stable state (already updated or not yet updated) when you run the build. That is, the time of the build system is well synchronized with the work of updates in network repositories.

 

14 hours ago, Igor said:

This can be done, but we will have more cached files. (this and that) desktop with this and that display manager. I am trying to avoid this situation. If possible. Let's leave this problem open. I will try another approach. 

IMHO I suggest for all versions of images where DE is used to use the standard approach from "adult" systems. The source image must use some kind of login Manager (for example, lightdm). This corresponds to the logic of safe operation and is a common (familiar) option for users moving from PC to ARM. The current state of Armbian and the hardware it runs on has already become quite Mature and capable of working on an equal footing with the PC. If the user wants to make a autologin or use "nodm" - he will do it yourself (good manuals for this is more than enough). In the server options of images, the default should not be any managers of the entrance.

 

15 hours ago, Igor said:

Another issue is u-boot. We will drop u-boot package installation for all boards. Only compiling, packing and writing. But we need to have "write_uboot_platform" definitions for nand-sata-install and/or manual install/update via armbian-config ...
 

In this case variable ADD_UBOOT is deprecated.

In addition. I would put the u-boot Assembly in a separate process, with mandatory testing for the correctness of the assembled version on the intended devices. And only after that place the packages in the repository for Assembly by others or for use by users. U-boot is an important element for the safe use of the device (this is especially true for devices with built-in memory, where it is very difficult to fix not correctly recorded in memory u-boot).

[balbes150]

On 12/11/2018 at 3:34 PM, Igor said:

The main idea is written here: https://github.com/armbian/build/tree/tvboxes/config/packages It's actually simple. Upgrade might be more problematic.

I took a quick look, Yes, the idea is good, to make the creation of all the necessary packages in a separate procedure and have a common source with the packages to install. I understand that it is now possible to gradually migrate her assembling the necessary packages for TV boxes ?

[Igor]

On 12/18/2018 at 8:55 AM, balbes150 said:

Yes, this also happens to users (and quite often). When they run the update and as a result, some of the features they used are broken (you have to remove conflicting packages).

This should not happen. At least not with a basic system packages and matured release base. At this point neither Stretch nor Bionic should not have such problems. In their beta stages, it could happen, yes. In any case, we can't afford to have our own relation set set. It would be good, but I am not sure if we can afford adding this complexity to already complex system? But we do have a workaround/hack - our repository, which can be used to insert a patched package - I am using it to overwrite broken upstream 32b Chromium build. One must prepare a package and push it here: https://github.com/armbian/upload 

 

On 12/18/2018 at 8:55 AM, balbes150 said:

IMHO I suggest for all versions of images where DE is used to use the standard approach from "adult" systems. The source image must use some kind of login Manager (for example, lightdm). This corresponds to the logic of safe operation and is a common (familiar) option for users moving from PC to ARM. The current state of Armbian and the hardware it runs on has already become quite Mature and capable of working on an equal footing with the PC. If the user wants to make a autologin or use "nodm" - he will do it yourself (good manuals for this is more than enough). In the server options of images, the default should not be any managers of the entrance.

You are suggesting changing defaults to lightdm? Well, we could do that. But let's leave this for second phase. 

 

On 12/18/2018 at 8:55 AM, balbes150 said:

In addition. I would put the u-boot Assembly in a separate process, with mandatory testing for the correctness of the assembled version on the intended devices. And only after that place the packages in the repository for Assembly by others or for use by users. U-boot is an important element for the safe use of the device (this is especially true for devices with built-in memory, where it is very difficult to fix not correctly recorded in memory u-boot).

U-boot is going out from the install and repository deployment part as well. It will be placed there after manual testings only. For all hardware, not just for TV boxes. Build and pack part doesn't need to be changed. 

 

On 12/18/2018 at 1:25 PM, balbes150 said:

I took a quick look, Yes, the idea is good, to make the creation of all the necessary packages in a separate procedure and have a common source with the packages to install. I understand that it is now possible to gradually migrate her assembling the necessary packages for TV boxes ?

Yes, it can be migrated. Do one family and test it.

[balbes150]

@Igor

I mean, a different situation.

I will describe in more detail on the example of the image Assembly. For example, I run a clean build (when I don't have a ROOTFS archive yet). The build system , in the process, receives all necessary packages from the network and creates a local archive with ROOTFS. After that, the build system additionally (separately) downloads the packages and dependencies for LIGHTDM installation and installs them in the temporary system generated for the image. After that, the build system forms the finished image. In this use case, receiving packets from the network for ROOTFS and receiving packets for the LIGHTDM Manager occurs at almost the same time, and it is unlikely that there will be a problem. After some time (for example, 3-5 days), I run the Assembly of the same image again. I already have a ready archive with ROOTFS and it is used (packages are not received from the network). But with further image building, the build system accesses the network and receives LIGHTDM packets (and its dependencies) from the network. For the time that has passed since the formation of the ROOTFS, you can change the part of the packet and a collision occurs. Adding to the build system, the process of updating packages on the build system , before installing LIGHTDM packages, can aggravate the situation with unsynchronization. The existing ROOTFS already has its own versions of packages and settings, and an attempt to update them automatically can give an unknown result. I suggest doing getting all the packets from the network reps that are used to form the ROOTFS and the login Manager at one time. That is, LIGHTDM packages and their dependencies must be part of the ROOTFS archive. You don't have to try to retrieve LIGHTDM packages from online repositories every time you run an image build.

[Igor]

4 minutes ago, balbes150 said:

@Igor

I mean, a different situation.

I will describe in more detail on the example of the image Assembly. For example, I run a clean build (when I don't have a ROOTFS archive yet). The build system , in the process, receives all necessary packages from the network and creates a local archive with ROOTFS. After that, the build system additionally (separately) downloads the packages and dependencies for LIGHTDM installation and installs them in the temporary system generated for the image. After that, the build system forms the finished image. In this use case, receiving packets from the network for ROOTFS and receiving packets for the LIGHTDM Manager occurs at almost the same time, and it is unlikely that there will be a problem. After some time (for example, 3-5 days), I run the Assembly of the same image again. I already have a ready archive with ROOTFS and it is used (packages are not received from the network). But with further image building, the build system accesses the network and receives LIGHTDM packets (and its dependencies) from the network. For the time that has passed since the formation of the ROOTFS, you can change the part of the packet and a collision occurs. Adding to the build system, the process of updating packages on the build system , before installing LIGHTDM packages, can aggravate the situation with unsynchronization. The existing ROOTFS already has its own versions of packages and settings, and an attempt to update them automatically can give an unknown result. I suggest doing getting all the packets from the network reps that are used to form the ROOTFS and the login Manager at one time. That is, LIGHTDM packages and their dependencies must be part of the ROOTFS archive. You don't have to try to retrieve LIGHTDM packages from online repositories every time you run an image build.

That's why nodm is a better choice  I really never had issues which you are talking about with it. I do build with lightdm very rare so I could not bump into those problems.  Well, one option is to add all Lightdm dependencies to the basic package base and install core package at the end. I will try if that makes any changes.

[balbes150]

18 minutes ago, Igor said:

That's why nodm is a better choice  I really never had issues which you are talking about with it. I do build with lightdm very rare so I could not bump into those problems.  Well, one option is to add all Lightdm dependencies to the basic package base and install core package at the end. I will try if that makes any changes.

Now Armbian is increasingly used as a full-fledged working system with DE (by the way, not only at home , but also in organizations). So it may work different users and absolutely necessary system of separation of entrance and lock when the lack in the workplace.

[Igor]

6 minutes ago, balbes150 said:

Now Armbian is increasingly used as a full-fledged working system with DE (by the way, not only at home , but also in organizations). So it may work different users and absolutely necessary system of separation of entrance and lock when the lack in the workplace.

I got that. I will try to fix this problem somehow. Do we have an realistic alternative to lightdm?

[balbes150]

3 minutes ago, Igor said:

I got that. I will try to fix this problem somehow. Do we have an realistic alternative to lightdm?

Perhaps lightdm is not the best solution and there are others, but it has been repeatedly tested on ARM and works correctly, has the ability to easily configure the auto-login.

[Igor]

4 hours ago, balbes150 said:

Perhaps lightdm is not the best solution and there are others, but it has been repeatedly tested on ARM and works correctly, has the ability to easily configure the auto-login.

Solved this way.
https://github.com/armbian/build/commit/a9f552f5ff527f905b2300f74c61f1f9e266cdf4

[gprovost]

@Igor Are you going to define a deadline when all the different board maintainers must have migrate&adapt whatever stuff they have in build/packages/bsp/ into build/config/packages/ ?

[Igor]

22 minutes ago, gprovost said:

@Igor Are you going to define a deadline when all the different board maintainers must have migrate&adapt whatever stuff they have in build/packages/bsp/ into build/config/packages/ ?

 

On 12/19/2018 at 9:17 PM, Igor said:

Yes, it can be migrated. Do one family and test it

I plan to invite more people when things are matured enough. Are they already? Perhaps. You are welcome to join as well, but from current perspective I have no idea when this will be merged upstream. Just move features over and keep them aligned at this stage.

If anyone wants to keep themselves busy: we need a detailed study how to replace/upgrade current packages to new ones.

[balbes150]

2 minutes ago, Igor said:

If anyone wants to keep themselves busy: we need a detailed study how to replace/upgrade current packages to new ones. 

Just like in Debian, add the full version number to the package name. When you run a shared build script, it checks the number in the description and if it has changed, it starts the build of a new variant of the package with the replacement (or not) of the current package with a new one.

 

 

[Igor]

17 minutes ago, balbes150 said:

Just like in Debian, add the full version number to the package name. When you run a shared build script, it checks the number in the description and if it has changed, it starts the build of a new variant of the package with the replacement (or not) of the current package with a new one.

I mean upgrade of current images. Can we expect problems?

[gprovost]

Is it the objective to address upgrade from previous bsp package to new bsp++ packages ?

 

If I grasped correctly the new approached, then I guess will need then to replace the current armbian-bsp package by a metapackage pointing to armbian-bsp.deb + family.deb + board.deb.

 

This should address the upgrade path... but might still create some tricky use cases.

[balbes150]

33 minutes ago, Igor said:

I mean upgrade of current images. Can we expect problems?

This solves the problem of automatic updating of existing systems for users. New packages are sent to the network repository and when the user starts the system, new versions will be detected (as far as I remember, you have the formation of an archive with packages and it can be used). Maybe you should create a standard Armbian network repository, add it to the APT list during the build and then everything will be done automatically by users (when the standard "apt update"procedure will be run). By the way, the same network repository can be used to build images (get Armbian packages from It).

 

 

[balbes150]

In the process of building test images, I constantly get an error message about the installation of the package "armbian-bionic". I looked at the log and there is a conflict with the package "linux-bionic-root-rockpro64". This is when building images for s9xx, rk3399-tv etc. Where does this dependence (in the build scripts of the packages I can't find instructions on it) ?

[Igor]

5 hours ago, balbes150 said:

In the process of building test images, I constantly get an error message about the installation of the package "armbian-bionic". I looked at the log and there is a conflict with the package "linux-bionic-root-rockpro64". This is when building images for s9xx, rk3399-tv etc. Where does this dependence (in the build scripts of the packages I can't find instructions on it) ?

Are you working with up2date script? I did some changes and this package (linux-bionic-root-rockpro64) is deprecated - is not building anymore. I am also not sure that everything is working properly. Still debugging.

[balbes150]

12 hours ago, Igor said:

Are you working with up2date script? I did some changes and this package (linux-bionic-root-rockpro64) is deprecated - is not building anymore. I am also not sure that everything is working properly. Still debugging.

I run an image build, and an error occurs during the image build process in the log.

 

armbian-config is already the newest version (5.67).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
dpkg-deb: building package 'armbian-bionic' in '/home/user/build/output/debs/bionic/armbian-bionic_5.67_arm64.deb'.
Selecting previously unselected package armbian-bionic.
dpkg: regarding .../armbian-bionic_5.67_arm64.deb containing armbian-bionic:
 linux-bionic-root-rockpro64 conflicts with armbian-bsp
  armbian-bionic provides armbian-bsp and is to be installed.

dpkg: error processing archive /root/armbian-bionic_5.67_arm64.deb (--install):
 conflicting packages - not installing armbian-bionic
Errors were encountered while processing:
 /root/armbian-bionic_5.67_arm64.deb

[Igor]

9 hours ago, balbes150 said:

Errors were encountered while processing:

This should be fixed now. Make sure you use PACKAGES_RECOMPILE="yes" or delete them prior to testing.

[balbes150]

On 12/25/2018 at 7:26 PM, Igor said:

This should be fixed now. Make sure you use PACKAGES_RECOMPILE="yes" or delete them prior to testing.

If I add an option PACKAGES_RECOMPILE="yes" , the build passes without errors, but the system does not start, no uInitrd is generated. Without this option (even with the error message in the logs) the image is built correctly and works.

It's not critical, it's information.

[Igor]

4 hours ago, balbes150 said:

no uInitrd is generated.

Fixed.

 

4 hours ago, balbes150 said:

PACKAGES_RECOMPILE="yes

Fixed second time  armbian-config from the repository was making troubles. Now with a new version it doesn't download it again.

Tests:
RockPro64 Bionic default http://ix.io/1wUW and Tinkerboard dev, Bluetooth and audio was tested http://ix.io/1wV8

[JMCC]

On 12/10/2018 at 10:37 PM, Igor said:

Welcome to try adding perhaps Tinkerboard MM script as a new multimedia/something package. What shall be its name? If some 3rd party packages needs to be placed to the repository, put them here: https://github.com/armbian/upload

You read my mind! I'm brushing up the last stages for the RK3328/3399 multimedia support, and after that I was going to propose to add it to the build script.

 

So my plan is to release the media scripts for those two SoC's in a few days, and after having some feedback from the community integrate the main features into the main build.

 

Edit: I mean including all three RK SoC's: 3288, 3399 and 3328.

[NicoD]

1 hour ago, JMCC said:

I'm brushing up the last stages for the RK3328/3399 multimedia support

Is this for 4K video for the RK3399? I've tried the newest FriendlyElec image but didn't have luck with this. I'll try the core image tonight. On the RockPi4B in Kodi in Recalbox 4K video's play well, but sound doesn't yet work there. 
Do you know if hardware encoding could be available for the RK3399's?

 

1 hour ago, JMCC said:

and after having some feedback from the community

If you need something tested, please let me know.

I'm now making a video about video editing on SBC's. The RK3399's are the best for this, but 1080p video preview isn't perfect now. So if this could be fixed, then this would be awesome news.
I should record my video this weekend, but I can wait with it if I could improve the experience.

Thanks for the great work, greetings.