Overriding new kernel versions scheme

[belegdol]

Hello,

 

the new kernel versioning scheme consists of the latest git tag and several hashes:

https://github.com/armbian/build/blob/main/lib/functions/artifacts/artifact-kernel.sh

With boards like odroid-xu4 for which the upstream git is only updated occasionally, this leads to two problems:

1. the generated version is lower than what is currenty getting shipped: `5.4.228-S3043-De511-P0a53-C0750H6842-Bb436` vs `23.02.2`

2. it cannot be guaranteed that the versions go up reliably: 5.4.232 is `5.4.228-S3043-De511-Pc02c-C0750H6842-Bec1c` and 5.4.233 is `5.4.228-S3043-De511-P0a53-C0750H6842-Bb436`

While I can understand using the kernel version than armbian version is better, this currently only works if every version is tagged. Is there a variable which I could use to get the old behaviour back? Thanks.

[Werner]

I think @rpardini can explain best what is happening there.

[belegdol]

I understand what is happening, the comment explains it well. I am merely asking if what is happening can be overridden.

[Igor]

Adjusting versioning is one of next major things before we will start to publish nightly packages again. I don't know if what we have now is correct / as designed. 
 

I am aware introducing / switching to NEXT it is painful for regular processes and all developers involved but we are getting many improvements on a long run.

[belegdol]

Would I get the old version scheme if I checkout v23.02 branch? I tried building with LIB_TAG=v23.02 but it still got me the new scheme:

./compile.sh BOARD=odroidxu4 BRANCH=current LIB_TAG=v23.02 BUILD_ONLY=kernel KERNEL_CONFIGURE=no USE_CCACHE=no KEEP_KERNEL_CONFIG=no DEB_COMPRESS=xz

 

[SteeMan]

@belegdol  To use the old code you want to checkout the 'master' branch of the build repository:  git clone --depth=1 --branch=master https://github.com/armbian/build

Note that that branch is no longer maintained, but you can use it to test the differences between old and new build functionality.

[SteeMan]

@belegdol On your first point, yes that is a one time issue as a result of using the kernel version.  While undesirable, I believe it is a necessary change.

On your second point, while I understand your point and the functionality has changed, it also didn't work the way you wanted under the old master branch.  In the master branch the .deb files end up with the same version number across builds, there is no incremental build number.  The version is the armbian version and thats all.  At least in the new main branch the version changes so each build (where there are changes) will at least get a different .deb file.  So I would say the new functionality is better, but still not ideal.

[belegdol]

I agree regarding the one-time change. I also agree that the old scheme was not ideal either. Having said that , with the old version I could either edit the VERSION file or use a variable (was it sublevel?) To get the versions to go up consistently. With the new versioning scheme this does not seem to be possible, saving for forking the hardkernel repo, tagging kernel versions myself and telling the compile script to use that instead.

[going]

@belegdol

Are you interested in how to do this in your local assembly? Or is it a problem with updates, when a new kernel with a lower version should be updated by apt to the old kernel from the armbian repository but with a higher version?

I want to understand the essence of the problem in a broader sense.

And information for everyone.
I am continuing to develop the master branch. At home locally.
And knowing, understanding what the user wants is very important for all of us.

[going]

03.03.2023 в 10:04, belegdol сказал:

`5.4.228-S3043-De511-P0a53-C0750H6842-Bb436` vs `23.02.2`

 

5.4.228 - The upstream kernel source code version. And that's right.

S3043-De511-P0a53-C0750H6842-Bb436 - It looks like some kind of hash.
@rpardini Ricardo, what information can I extract from this?
For example, a user reports some kind of error in the kernel and sends me a package file. I see this information.
Can I reproduce the build of this particular kernel to understand the cause of the error?

[belegdol]

@going  it is predominantly about local assemblies. I often update kernels to a new point release and knowing which package is the newest one helps a lot. Not having local update overwritten by the distributed one is a welcome bonus but i understand that there is going to be some churn when switching from one scheme to another.

[going]

1 час назад, belegdol сказал:

it is predominantly about local assemblies

Do you build the kernel yourself and update it?
Did I understand correctly? And do you want your versions to run up?

[belegdol]

Yes:

https://github.com/armbian/build/pulls?q=is%3Apr+author%3Abelegdol+is%3Aclosed

The issue of versions not going up consistently is bound to affect official packages as well. Until hardkernel tag a new release, the kernel version is not going to change. Patches hash (Pxxxx) will change, but hashes cannot be guaranteed to go up.

Until now I was adding another element to the VERSION file (23.02.2.1, 23.02.2.2, etc.). This way my local builds will not get updated by the system 23.02.2 unless armbian shipped a larger update.

[SteeMan]

1 hour ago, belegdol said:

Until now I was adding another element to the VERSION file (23.02.2.1, 23.02.2.2, etc.). This way my local builds will not get updated by the system 23.02.2 unless armbian shipped a larger update.

You can do the equivalent by having a patch to the kernel Makefile that does the same thing to it's version number

[going]

03.03.2023 в 10:04, belegdol сказал:

it cannot be guaranteed that the versions go up reliably: 5.4.232 is `5.4.228-S3043-De511-Pc02c-C0750H6842-Bec1c` and 5.4.233 is `5.4.228-S3043-De511-P0a53-C0750H6842-Bb436`

 

1 час назад, belegdol сказал:

The issue of versions not going up consistently is bound to affect official packages as well.

But you apply a lot of patches that change the kernel version in the Makefile.

If I can judge by this code, the bicycle has been invented here again.

 

Please show the output of the command for the assembled package:

dpkg --info output/debs/linux-image-*.deb

 

[belegdol]

$ dpkg --info output/debs/linux-image-*.deb
 neues Debian-Paket, Version 2.0.
 Größe 28852376 Byte: control-Archiv= 1284 Byte.
     710 Byte,    10 Zeilen      control              
     807 Byte,    16 Zeilen   *  postinst             #!/bin/bash
     612 Byte,    13 Zeilen   *  postrm               #!/bin/bash
     891 Byte,    21 Zeilen   *  preinst              #!/bin/bash
     608 Byte,    13 Zeilen   *  prerm                #!/bin/bash
 Package: linux-image-current-odroidxu4
 Version: 5.4.228-S3043-De511-P2c81-C951fH6842-Bb436
 Source: linux-5.4.234
 Architecture: armhf
 Maintainer: Igor Pecovnik <igor.pecovnik@****l.com>
 Section: kernel
 Provides: linux-image, linux-image-armbian, armbian-current
 Description: Armbian Linux current kernel image version "5.4.228" git revision "3043e57d9fe81d5031c98da852e84e7956b4a953" codename "Kleptomaniac Octopus" drivers hash "e511b9159daace9f" patches hash "2c81c7c1a0a42d1f" .config hash "951f7fdd7fb40928" .config hook hash "68429bb5c7c735ac" framework bash hash "b436f1853a32cf25"
  This package contains the Linux kernel, modules and corresponding other
  files, kernel_version_family: 5.4.234-odroidxu4.

With the new scheme the package version appears to come from the git tag, not from the makefile. I am applying a lot of patches to change the kernel version because hardkernel only update their tree irregularly. If you check the git history of the patches folder, this is what the other maintainers have been doing before as well:

https://github.com/armbian/build/commits/main/patch/kernel/archive/odroidxu4-5.4

[going]

38 минут назад, belegdol сказал:

Version: 5.4.228-S3043-De511-P2c81-C951fH6842-Bb436
 Source: linux-5.4.234

5.4.234 - Makefile

There is only one way to change this.
Change the function itself.

Somewhere here it is necessary to redefine the artifact_version variable.

 

It is important to understand that you are on your own in this.
I can only give "harmful" advice.

[Igor]

What if revert versioning change to what we have now? And return to this issue later, when we have a system in better stage. We need to make upgrade possible and nothing has been done in this concern. Its not just about upgrading our builds, but also builds people generate for themselves by using a script. They / we can only wait awhile if some critical component is not functioning as expected. (even wrong way)

 

 

[going]

2 часа назад, Igor сказал:

What if revert versioning change to what we have now?

And what do we have now?

2 часа назад, Igor сказал:

We need to make upgrade possible and nothing has been done in this concern.

Updating for the end user from the armbian repository with a downgrade of the package version is solved very simply at the level of the repository itself.

A description of how to do this exists in the debian documentation.

 

18 часов назад, belegdol сказал:

Version: 5.4.228

This version is in the Makefile before patches are applied.

18 часов назад, belegdol сказал:

Source: linux-5.4.234

This version is after applying patches.

 

The package must contain an up-to-date one.
The problem is that I can't see where a miss occurs in the script.

I can't do anything to help. Therefore, I just advise you.

[Igor]

26 minutes ago, going said:

Updating for the end user

We need to provide this automatically for user.

[going]

@Igor Repository settings occur in this function.

rootfs/apt-sources.sh

But it could be a package named armbian-update-${variant}.deb
In this package, we can configure repository priorities and prescribe specific dependencies.
But before that, the packages must be assembled correctly.

[Igor]

14 hours ago, going said:

But it could be a package named armbian-update-${variant}.deb
In this package, we can configure repository priorities and prescribe specific dependencies.
But before that, the packages must be assembled correctly.

 

It looks like we are in the middle of this transition. Some packages are in old schema (bsp), some new (kernel, dtb ...) 

 

I understand we need to generate upgrade package, but all this needs to be adjusted and well tested. What I propose is to go (temporally) back to the original versioning scheme so we can start producing nightly images and upgrade will just work.

If we are o.k. (Richardo is fine with) this is best what can be done in short time, can you help in resolving this?

[going]

1 час назад, Igor сказал:

What I propose is to go (temporally) back to the original versioning scheme so we can start producing nightly images and upgrade will just work.

The problem has arisen here and now at this moment in time. And it must be solved here and now.
It exists for the "main" branch. It exists in my branch. And it's the same thing. A small difference in details will not play any role.

I am against going back to the old version system.
It creates more problems than it allows you to do with minimal actions.

We have to do everything at once and now.

To begin with, let's list the currently existing repositories.

 

Скрытый текст

leo@bananapim64:~$ cat /etc/apt/sources.list
deb http://ports.ubuntu.com/ jammy main restricted universe multiverse
#deb-src http://ports.ubuntu.com/ jammy main restricted universe multiverse

deb http://ports.ubuntu.com/ jammy-security main restricted universe multiverse
#deb-src http://ports.ubuntu.com/ jammy-security main restricted universe multiverse

deb http://ports.ubuntu.com/ jammy-updates main restricted universe multiverse
#deb-src http://ports.ubuntu.com/ jammy-updates main restricted universe multiverse

deb http://ports.ubuntu.com/ jammy-backports main restricted universe multiverse
#deb-src http://ports.ubuntu.com/ jammy-backports main restricted universe multiverse

leo@bananapim64:~$ cat /etc/apt/sources.list.d/armbian.list 
deb [signed-by=/usr/share/keyrings/armbian.gpg] http://apt.armbian.com jammy main jammy-utils jammy-desktop

leo@bananapim64:~$ tree /etc/apt
/etc/apt
├── apt.conf.d
│   ├── 01autoremove
│   ├── 01-vendor-ubuntu
│   ├── 02-armbian-compress-indexes
│   ├── 02-armbian-periodic
│   ├── 02-armbian-postupdate
│   ├── 70debconf
│   ├── 71-armbian-no-recommends
│   └── 81-armbian-no-languages
├── auth.conf.d
├── keyrings
├── preferences.d
├── sources.list
├── sources.list.d
│   └── armbian.list
└── trusted.gpg.d
    ├── ubuntu-keyring-2012-cdimage.gpg
    └── ubuntu-keyring-2018-archive.gpg

===============
leo@bananapim64:~$ apt list --upgradable
.....
armbian-bsp-cli-bananapim64/jammy 23.02.2 arm64 [может быть обновлён с: 23.02.0-trunk]
armbian-firmware/jammy,jammy,jammy 23.02.2 all [может быть обновлён с: 23.02.0-trunk]
armbian-plymouth-theme/jammy,jammy,jammy 23.02.2 all [может быть обновлён с: 23.02.0-trunk]
.....
linux-dtb-current-sunxi64/jammy 23.02.2 arm64 [может быть обновлён с: 6.1.14-Armbian.23.02.0]
linux-image-current-sunxi64/jammy 23.02.2 arm64 [может быть обновлён с: 6.1.14-Armbian.23.02.0]

 

 

I see only one armbian repository.
And an empty preferences.d directory.

 

You should make a new repository named jammy-staging for packages with a downgraded version of packages and configure repository priorities in preferences.d/80-armbian-staging.pref.

	cat <<- 'EOF' > "${basedir}"/etc/apt/preferences.d/80-armbian-staging.pref
		Package: *
		Pin: release n=jammy-staging
		Pin-Priority: 1001
	EOF
	cat <<- EOF >> "${basedir}"/etc/apt/sources.list.d/armbian.list
		deb [signed-by=/usr/share/keyrings/armbian.gpg] http://apt.armbian.com jammy-staging main
	EOF

You may want to choose a different priority policy, for example, only for certain packages.
Read also man apt_preferences
 

 

[going]

3 часа назад, Igor сказал:

If we are o.k. (Richardo is fine with) this is best what can be done in short time, can you help in resolving this?

Today I am engaged in putting the code in order with the assignment of the kernel version for the master branch.

After the code is stabilized, I will post it in my repository.

Ricardo will be able to independently apply this logic to his code in the main branch

[rpardini]

Hello folks, sorry for the absence, I've been facing connection issues.

Let me try and explain a bit, from the beginning. I will focus on kernel, but most applies also to u-boot, firmware, etc; also keep in mind that not all packages have been migrated to the new scheme: BSPs, Desktop BSPs, etc, are not yet changed (too many things to hash / little benefit / too many things to decide before that).

 

The reason we've a new version scheme is for consistent caching and fast CI runs.

The important part about the caching is that we need to obtain the "desired" version of the kernel, **before** actually git clone/fetching the sources, before patching, and before compiling; otherwise it's useless: if the user/developer has to do any/all of this to get to the version, then it's "too late" for caching. This needs to be crystal clear, otherwise everything just looks like an insane person did random stuff and broke Versioning for no reason.

 

Important: the caching part benefits some users/builders, in the sense that someone casually building an image at home might hit the cache, but the main target/reason it exists is for our (Armbian) own Continuous Integration (CI): every PR, every nightly, every release, etc, should benefit from the caching, but it has to be consistent (not use wrong version), and hit ratios should be high. Before this scheme, we had only REPOSITORY_INSTALL mechanism, which frequently installed wrong/mismatched versions, we just said "use whatever is in the apt repo", which could be literally anything, and frequently led to half-working images.

 

Also, "apt repo" is a very inflexible/demanding solution in terms of infrastructure, which requires HTTPS servers/mirrors, Release lists, etc, and is not available for free for every developer/partner. New caching uses standard OCI registries, which are available for anyone to host, and even hosted for free on GitHub (ghcr.io) and DockerHub among others.

 

That said, let's get to how we calculate the version for the kernel package. The code, containing a long explanation, is at https://github.com/armbian/build/blob/main/lib/functions/artifacts/artifact-kernel.sh - but in essence goes like this:

 

1) "base_version" (bv): Get the KERNELSOURCE and KERNELBRANCH variables (usually set in family config file for the board being built). Use `git ls-remote` to obtain the SHA1 of the commit of that. Obtain the HTTP URL for the Makefile at that commit. ("View raw" in GitHub web interface, but also for GitLab, and others). Get the Makefile contents (using HTTP, not Git, since there is not "partial git fetch"). Parse the Makefile for the version information inside, including the "codename". (Codename here is just to "prove" that data comes from Makefile, not "git tag"). Cache all this on disk for a few minutes.

2) "S"ha1: just the SHA1 of the commit mentioned above. 

3) "D"rivers: Obtain a SHA256 hash of the "drivers". Kernel drivers (EXTRAWIFI etc) are just a complex variation of patches; they are shared across families, and are not stored as patches, but in the end are transformed into one huge patch, so behave like one.

4) "P"atches: Obtain a SHA256 hash of the "patches": simple hashing of the patches/series/etc under KERNELPATCHDIR (and userpatches variations). If no patches, "0000" is used.

5a) "C"onfig: Obtain a SHA256 hash of the ".config" file that is gonna be used for the build. LINUXCONFIG (and userpatches variations).

5b) "H"ooks: Obtain a SHA256 hash of the hooks that might modify the .config. (This is a bit more complex to explain, since it involves extension mechanism, which is out of scope here). Suffice to say that if you change a hook that changes kernel configuration, the version should change.

6) "B"ash: Some lib/framework code, if changed, should cause the version to change as well. Not all bash code, but bash that calls the "make", or does the packaging, if changed, need to change the version. Currently "lib/compilation/kernel*.sh" is hashed.

 

Hopefully, you can now understand that `Version: ` below. Each SHA256 hash is shortened to a mere 4 characters:

 

On 3/5/2023 at 3:58 PM, belegdol said:

Version: 5.4.228-S3043-De511-P2c81-C951fH6842-Bb436

 

 

So now we've to address two things: a) the original question by @belegdol; b) the general question of "how to control upgrades via apt".

 

------------------------------------------------------------------------

 

Ref the "odroidxu4" situation: @belegdol maintains that kernel family, in the following way (please correct me if I'm wrong):

a) Vendor, HK, has an  Git repo, which has a 5.4.y version, plus HK/Exynos patches. That Makefile currently says "5.4.228". It has heavily lagged behind official releases in the past AFAIK but seems pretty recent recently. HK "merges" updates from mainline, instead of cleanly rebasing, though, since they've lost all hope of mainlining their stuff and merging is easier.

b) Belegdol extracts new/recent 5.4.y patches from upstream/kernel.org, (possibly?) manually fixes them so they apply cleanly on top of HK kernel, adds them to Armbian via PRs. Those patches modify the Makefile so the version is incremented.

 

So AFAIK this is the only kernel family that is done this way (we did some similar stuff for rk3588 legacy, but thankfully no more). Every other family works in reverse: we start with a pristine/clean/updated kernel.org branch (whose Makefile is updated), and then patch what is needed on top of that. That way the kernels "auto update", unless some patch breaks.

 

Hopefully it is clear, together with the above explanation, why the  "wrong" version is used: only the original Git's Makefile version (not tag... but tag's Makefile's contents) is considered, since patches are only ever applied after Version is calculated, and cache possibly hit or missed.

 

So while I confess I had not xu4 as the primary target of this whole thing, it representing less than 1% of all boards. Still, there needs to be a way to make it work, we just need to decide how.

 

One possibility: invert the XU4 patching scheme, by rebasing HK's branch onto an upstream tag, and extracting the patches (git format-patch); then change our KERNELSOURCE/KERNELBRANCH to mainline, and let it auto update. This might, or not, be a lot of work initially, and might, or not, impose a lot less work over time. It depends on how many changes there are in HK Git, if the relevant patches are constantly updated, etc. From what I hear HK has little work done on the Exynos, but I am not an expert. I would like this scheme since it would align XU4 with the rest.

 

Another possibility: introduce a way to (optionally) override the "base_version" (example, variable FORCE_KERNEL_EXACT_VERSION_VIA_PATCHES="5.4.232") directly in the family configuration, use that for the artifact version if present, and change nothing else. This is _more_ work for xu4 maintainer, and might be get out-of-sync with reality (change patches, but not change variable, and vice versa). This might be useful also for other situations, although none come to mind right now, it might be useful for Igor's "emergencies" 😉

 

------------------------------------------------------------------------

 

Now about "how to control upgrades via apt". 

 

To understand this, we need to understand the formally defined Debian Policy about the "Version: " field in the DEBIAN/control file.

I recommend we all read https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-version -- it is one of the more dense, complex, and important paragraphs of text ever written by Debian, and defines how apt decides what version is "higher" than another.

There, we can see there are 3 components, each with its own rules, and the special rule about the "~" (tilde) character.

 

The important part: I did not want to "impose" the final apt versioning scheme before we talked about it.

Also: there's pending work in the general area of "repository management/publishing" that needs to be done before we can publish _anything_, consistent or not, to any apt repository. 

I was told we had until May/2023 (so circa 2 months) to reach consensus and implement those -- but now it seems the sky is falling down much earlier?

 

So current "Version: " is a non-working, non-deterministic, inconsistent situation in regards to apt upgrades: if you try to predict how our "bv+S+D+P+CH+B" will be handled by apt, you will soon realize it's inconsistent: "fixed"/later patches on same base version might get downgraded to broken ones, and vice versa, almost randomly (as is the property of SHA256 hashes).

 

One extremely easy way to "fix it" is to prefix our current "bv+S+D+P+CH+B" with the $REVISION (which comes from VERSION file or equivalent userpatches) and a tilde.

That would force apt to respect the "old" versioning scheme, while still keeping caching consistent.

Unfortunately, doing that would drastically decrease cache hit ratio, since changing VERSION file would cause an otherwise identical kernel to be ignored/rebuilt. That might happen because user/dev changed userpatches/VERSION, but is otherwise identical to main, or because we branched off a release/tag (otherwise identical to main at that instant).

 

But it might be the best option right now, especially if no-one else is willing to invest brain time on this. I could then twist my own brain on how to recover the cache hit ratio somehow else (eg, search cache for both with-$REVISION and without-$REVISION variants, invent some "IS_RELEASE=1" variable, doing GitHub Actions acrobatics, etc; is a lonely world where only Igor and Ricardo live).

 

Similar option: transforming $REVISION ("23.05.01") into an integer (230501) and using that as the "epoch" part of Version field, would also work, but same cache-non-hitness applies.

 

-----

 

Phew, that was long. If you read all this, give yourself a pat in the back. Let me know what you think on all the different points... @going @Werner @belegdol @Igor @SteeManand everyone else interested.

 

Edited March 8, 2023 by rpardini
forgot the "S"ha1 part...

[Werner]

1 hour ago, rpardini said:

But it might be the best option right now,

Agreed. Better hit a bit of cache than not at all. So still a win if I understood everything else correctly.

[belegdol]

@rpardini, thank you for the detailed explanation and for clarifying where 5.4.228 is coming from. I incorrectly assumed it comes from the git tag.

As far as odroidxu4 is concerned, your description of the current update process is correct. I was not aware it is an exception in terms of how this kernel is kept up-to-date. Knowing this, I think my preference would be to invert the patching order as well. Please note that I have not tried it yet. Having said that, almost all of the point release updates did not require manual fixing, which gives hope that rebasing could potentially work once initially set up.

The solution with the variable would only realistically fly if we can generate the said variable automatically. Otherwise, someone is bound to forget to update both at some point leading to misleading version numbers.

[belegdol]

I have now checked the rebase path and there are over 14k commits to go through. At 142nd commit I have arrived at 4th merge conflict. At this rate there are going to be 400 merge conflicts to go through. Given that my knowledge of C is non-existent, I think we might need a new plan.

[rpardini]

41 minutes ago, belegdol said:

I think we might need a new plan.

 

Yes. I've remembered that adeepv/adeepn (Viacheslav) had something related: a privately hosted, pure-git kernel, to which we couldn't get HTTP access at all. In that case, including the base_version (eg v6.2.1) is not possible. Both for that case, and for this "HK's 5.4.y xu4 + incremental patches", we might introduce something like "KERNEL_SKIP_MAKEFILE_VERSION=yes" to be configured in family file (and _not_ FORCE_KERNEL_EXACT_VERSION_VIA_PATCHES like I mentioned before). Together with the "add $REVISION prefix to all artifact versions", we'd just not have the 6.2.1 version component, and instead just use the "S"HA1 of the commit (which is always available). This should also avoid the "forgot to update" misleading problem we've identified above. And it solves 2 problems...

[going]

20 часов назад, rpardini сказал:

Let me try and explain a bit, from the beginning. I will focus on kernel,

This detailed article description requires reflection.

20 часов назад, rpardini сказал:

One possibility: invert the XU4 patching scheme, by rebasing HK's branch onto an upstream tag, and extracting the patches (git format-patch); then change our KERNELSOURCE/KERNELBRANCH to mainline, and let it auto update. This might, or not, be a lot of work initially, and might, or not, impose a lot less work over time. It depends on how many changes there are in HK Git, if the relevant patches are constantly updated, etc. From what I hear HK has little work done on the Exynos, but I am not an expert. I would like this scheme since it would align XU4 with the rest.

This is a very good way. The initial extraction will be difficult, but in the following years it will allow us to avoid a lot of work.

But the problem is hidden somewhere else.
 

I don't see much difference in the way we came to the state in the working directory of the kernel. In what order we applied the patches. One way or another.

The algorithm will miss for some reason. It will fly past the target. This needs to be sorted out.

I'll say more. Some patches do not change the values of variables in the Makefile, but add changes that will eventually add a local version.

That is, in the process of the make command.

For example, the PREEMPT_RT patch (one of the five known to me) or the user can add a local version during kernel configuration..