Overriding new kernel versions scheme

[going]

18 часов назад, rpardini сказал:

To understand this, we need to understand the formally defined Debian Policy about the "Version: " field in the DEBIAN/control file.

I recommend we all read https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-version -

I will vote with both hands for us to adhere to this policy.

 

[going]

19 часов назад, rpardini сказал:

The code, containing a long explanation, is at https://github.com/armbian/build/blob/main/lib/functions/artifacts/artifact-kernel.sh - but in essence goes like this:

 

1) "base_version" (bv): Get the KERNELSOURCE and KERNELBRANCH variables (usually set in family config file for the board being built). Use `git ls-remote` to obtain the SHA1 of the commit of that. Obtain the HTTP URL for the Makefile at that commit. ("View raw" in GitHub web interface, but also for GitLab, and others). Get the Makefile contents (using HTTP, not Git, since there is not "partial git fetch"). Parse the Makefile for the version information inside, including the "codename". (Codename here is just to "prove" that data comes from Makefile, not "git tag"). Cache all this on disk for a few minutes.

2) "S"ha1: just the SHA1 of the commit mentioned above. 

3) "D"rivers: Obtain a SHA256 hash of the "drivers". Kernel drivers (EXTRAWIFI etc) are just a complex variation of patches; they are shared across families, and are not stored as patches, but in the end are transformed into one huge patch, so behave like one.

4) "P"atches: Obtain a SHA256 hash of the "patches": simple hashing of the patches/series/etc under KERNELPATCHDIR (and userpatches variations). If no patches, "0000" is used.

5a) "C"onfig: Obtain a SHA256 hash of the ".config" file that is gonna be used for the build. LINUXCONFIG (and userpatches variations).

5b) "H"ooks: Obtain a SHA256 hash of the hooks that might modify the .config. (This is a bit more complex to explain, since it involves extension mechanism, which is out of scope here). Suffice to say that if you change a hook that changes kernel configuration, the version should change.

6) "B"ash: Some lib/framework code, if changed, should cause the version to change as well. Not all bash code, but bash that calls the "make", or does the packaging, if changed, need to change the version. Currently "lib/compilation/kernel*.sh" is hashed.

Thanks for the explanation!
The problem is in point 1)

 

We have to fill in the working directory of the future package in one way or another (extract the git state, apply patches, etc.).

We have to form files in the debian directory. Then only the "make" command works. All scripts for building debian packages are a kind of wrapper for this command.
The version should be formed at this stage and in the manner recommended in the debian documentation.

 

How to make the correct update when downgrading the version from the point of view of apt, it is written in the debian documentation. No brainstorming is needed for this.

[going]

void

[belegdol]

On 3/9/2023 at 8:32 AM, going said:

This is a very good way. The initial extraction will be difficult, but in the following years it will allow us to avoid a lot of work.

As I mentioned above, it is really a lot of work:

On 3/8/2023 at 7:30 PM, belegdol said:

I have now checked the rebase path and there are over 14k commits to go through. At 142nd commit I have arrived at 4th merge conflict. At this rate there are going to be 400 merge conflicts to go through.

I neither have the time nor the skill to do this unfortunately.

[Igor]

I had a discussion with Oleg about packaging. He got rid of DT package long time ago, while we kept it for "don't touch it if its not broken" reasons. Since we already break everything apart and we need to adjust versioning, repository might also need to be changed ... should we get rid of the DT package and place DT blobs to the same /boot/ location as in DT package (as done in media kernel)

 

 

[rpardini]

2 hours ago, Igor said:

should we get rid of the DT package and place DT blobs to the same /boot/ location as in DT package (as done in media kernel)

 

Well this is a completely different subject, but yeah, valid. That probably/finally explains why media had double-patches to mkdebian/builddeb (which was already patched/replaced by Armbian before patches applied). I was trying to understand that, for months, and failed. Not a single comment explained it, as seems to be Oleg's policy. I will revisit it now that I know wth is going on and maybe it will make sense, but sincerely...

 

Currently and before:

- linux-image: has kernel, modules, and DTs (!), (in /usr/lib/<kernel_id>) -- as expected by standard Debian tooling (u-boot-menu, flash-kernel, etc). We had a long discussion about this, 2 years ago. Did we forget? (No, it was just Oleg said he agreed, but ignored it and patched his own stuff "secretly" sans any explanation or discussion).

- linux-dtb: has DTs in /boot (as expected by "most" Armbian families, using u-boot and bootscripts, also the armbian-install, config, overlays, etc).

 

Either way, we can definitely accommodate this "single pkg" need, but I'd like to do it in a way that is 1) comprehensible 2) usable by more families than just one 3) not a maintenance nightmare, eg, NOT double-patching. I don't think that is unreasonable.

 

So if we could spell out what is the need, it would be easier... again: let's work together. we've 2 months to figure out stuff before release. 

I understand now that it seems that I just blindly changed the way `media` works and am crazy... might even be true, but, we can fix, and would already have fixed if more engineer talk and less drama. IMHO.

[SteeMan]

I am about to comment on things I don't know nearly enough about to offer good suggestions, but here I go anyway.

 

There are a lot of different issues tied up together in this thread and each person commenting is focused on different issues.  While it is true they all are valid issues, I think we are talking past each other some of the time.

 

1) First I think is the original issue of the thread dealing with the fact that the "xu4 + incremental patches"  builds are not detecting the kernel version number correctly.  And a proposal has been put forth to handle/workaround this corner case.

 

2) Second, much of the discussion has been on the deeper issues of overall package structure (issues that extend back to the old build framework as well).  These issues still need more discussion.

 

3) Third, is a new issue introduced by the new artifact caching strategy.   How to cache the artifacts in a way that provides maximum reuse.

 

Due to the current implementation of including the cache hash in the name of the .deb, the last two issues become tied together.  But I'm not sure they need to be.  What we need is a way to use both a local and remote cache to find the .deb's.  There are other ways to do this than including the hash in the file name. 

 

For example on the local cache the hash could be the parent directory name.  So instead of:

../debs/linux-image-current-meson64_6.1.13-S1ac8-De511-Pb8ce-C072bHfe66-B60c8_arm64.deb

it could be:

../debs/S1ac8-De511-Pb8ce-C072bHfe66-B60c8/linux-image-current-meson64_6.1.13_arm64.deb

This leaves the .deb package format/name independent from the caching mechanism.  Thus separating issues 2 and 3 above.

Yes it does mean that it is more difficult to find the .debs you build locally.  That may be an acceptable tradeoff to gain the new caching functionality, but that could be alleviated by creating soft links/hard links in ../debs to the files as they are created.  (Currently as there could be multiple different .deb's with the same name, in different hash directories.  This is due to "changes" that don't involve a kernel version number bump.  In this case the ../debs directory would have a link to the last version created, which essentially mirrors the behavior of the old master code)

For the remote cache the cache key could be created in a similar manner that separates the hash and filename as well.

 

Then we still have to sort out the design surrounding issue 2, but I think that discussion becomes simpler (but still very complex) than is currently is, with trying to squeeze our artifact cache on top of the debian packaging.

 

 

 

 

 

[rpardini]

On 3/11/2023 at 9:10 PM, SteeMan said:

For example on the local cache the hash could be the parent directory name.  So instead of:

../debs/linux-image-current-meson64_6.1.13-S1ac8-De511-Pb8ce-C072bHfe66-B60c8_arm64.deb

it could be:

../debs/S1ac8-De511-Pb8ce-C072bHfe66-B60c8/linux-image-current-meson64_6.1.13_arm64.deb

This leaves the .deb package format/name independent from the caching mechanism.  Thus separating issues 2 and 3 above.

 

Hmm, not really. Separating the local cache only solves the problems involved in local caching, but the real hard part is solving for repo (apt repo, apt.armbian.com and beta variant).

That simply can't be separated by folders, and usually only a single version of any given package is allowed in any single repo.

Thus the `Version: ` field (which determines the .deb filename too) is the bottleneck here, and directory tricks won't work.

But yeah, I very highly appreciate your try to detangle this stuff -- it is very, very hard.

 

I am moving on with the prefixing implementation, and to hell with cache hit ratio, at least for now -- we've many things breaking cos an old version from repo is being considered "newer" than a just-built local .deb. (eg: installing armbian-config brings in sunxi-tools which causes an "upgrade" to linux-image-sunxi-edge which is actually older than just-built. it's insane.)