Shallow Kernel Git trees via GitHub Actions

rpardini · May 24, 2022

Hello, I've finally gotten around to writing the GitHub Actions based Linux Kernel Shallow git trees exporter.

It's very simple, a single shell script and GHA workflow.

It prepares shallow bundle (and ready-to-go .tar too) everyday on a schedule and publishes to GitHub Releases.

Please take a look https://github.com/rpardini/armbian-git-shallow - theres a README that shows how to use the bundles outside of Armbian.

If this is a good idea, I'd move it to armbian/kernel-git-shallow in GitHub.

And then I'd start using those in armbian-next first as a test ground.

I mention I few people I know are interested in the subject @Igor and @going but of course everyone welcome to pitch in

Thanks

(PS: No idea why I'm required to tag a board for this post)

Werner · May 24, 2022

1 hour ago, rpardini said:

(PS: No idea why I'm required to tag a board for this post)

tagging is kind a stupid. the lesser of two evils was to simply enforce tagging in general. However I can disable entirely for particular forums. I guess this could be one of those.

going · May 24, 2022

https://github.com/rpardini/armbian-git-shallow/blob/6398b50d9ab1fbf9856f93203e10221e97903148/shallow_kernel_tree.sh#L23

This link shows that the file size to download is 2.4 GB.

Question. Do we have to do this every time it starts ."github/workflows/*" ?

Heisath · May 24, 2022

I'd guess this https://github.com/rpardini/armbian-git-shallow/blob/6398b50d9ab1fbf9856f93203e10221e97903148/shallow_kernel_tree.sh#L49 takes care of repetitive downloads.

rpardini · May 24, 2022

Thanks for the attention guys. I guess I did not explain this correctly...

The shallow_kernel_tree.sh script is run on GitHub Actions (not any user machine, ever), it downloads a lot of stuff (including a 2.4GB bundle, indeed), updates it, massages it into shallow, and publishes the output.

The output is here: https://github.com/rpardini/armbian-git-shallow/releases/tag/latest

Each kernel version gets a shallow bundle (file) around 250mb each.

That output would then (later) be used by armbian/build to seed a kernel tree, downloading from GH CDNs via HTTPS, instead of pulling everything from kernel.org's (or mirrors) git server.

After seeding, pulling from the git server is very fast / small download (a few megabytes).

That way people or CI servers building Armbian will only download those 250mb + a few.

Combining all this allows us to have a small, shallow git tree, that has the tags we need, and can be kept updated cheaply, without ever fetching/cloning it using shallow, and thus does not put any load on the git server.

rpardini · May 24, 2022

The reference for why this was written: https://github.com/armbian/build/pull/3476#issuecomment-1042063299

(mricon is from the Linux Foundation, who appeared after our CI was banned from kernel.org due to too much shallow fetching)

Also https://github.com/armbian/build/pull/3476#issuecomment-1042322996

going · May 25, 2022

15 часов назад, rpardini сказал:

That output would then (later) be used by armbian/build to seed a kernel tree, downloading from GH CDNs via HTTPS, instead of pulling everything from kernel.org's (or mirrors) git server.

After seeding, pulling from the git server is very fast / small download (a few megabytes).

That way people or CI servers building Armbian will only download those 250mb + a few.

This is what can be confidently called the most correct part of the algorithm.

15 часов назад, rpardini сказал:

The output is here: https://github.com/rpardini/armbian-git-shallow/releases/tag/latest

You've already done everything! Well done!

This is something that I am happy to store in a local folder.

And the ability to extract from the archive to the target folder cache/sources/linux-mainline.

This algorithm will greatly save free space inside the virtual machine.

But then the process of creating these archives in the build system is not necessary. It becomes a simple periodic manual work. Update this storage once a month.

rpardini · May 26, 2022

21 hours ago, going said:

But then the process of creating these archives in the build system is not necessary. It becomes a simple periodic manual work. Update this storage once a month

Yes. GitHub Actions workflow is setup to run everyday at 3am UTC. https://github.com/rpardini/armbian-git-shallow/blob/main/.github/workflows/main-latest.yml#L6

If you look at the GHA logs it is already working, everytime I push a commit but also everyday at 3am by itself: https://github.com/rpardini/armbian-git-shallow/actions

👍

-----

Apart from that, I am still investigating some vendor kernels (HardKernel, but also others) specially in 4.19 series, if I pre-seed the local copy with linux 4.19 shallow bundle, and then pull from that vendor's kernel tree, it comes down a lot (1.5Gb).

I suspect that is because the vendor started his kernel Git from somewhere _way_ before 4.19-rc1, and there is a mismatch during git negotiation for the fetch, and ends up pulling a lot.

@going I think you also regularly pull from "megous" repository that would be interesting to test too.

------

Another possible problem are developers/users in China which can't get to github.com for downloads.

For this case we could publish, in addition to GH Releases, also to some other FTP or rsync that we already have @Igor?

Igor · May 26, 2022

2 hours ago, rpardini said:

For this case we could publish, in addition to GH Releases, also to some other FTP or rsync that we already have @Igor?

Yeah, this way, just into some other folder:
https://github.com/armbian/scripts/blob/master/.github/workflows/pack-debian.yml#L230-L241

You need to move repository under /armbian in order that key works.

going · May 30, 2022

26.05.2022 в 12:29, rpardini сказал:

I suspect that is because the vendor started his kernel Git from somewhere _way_ before 4.19-rc1, and there is a mismatch during git negotiation for the fetch, and ends up pulling a lot.

@going I think you also regularly pull from "megous" repository that would be interesting to test too.

Today I have no problems with this.

The existing algorithm for the sunxi build line works well for repositories "https://github.com/", "https://source.denx.de/", "https://git.kernel.org/".

But it doesn't work for "https://kernel.googlesource.com/".

hege · June 8, 2022

Pardon me for asking silly questions, but why doesn't armbian just maintain one master clone of the upstream git repository with branches set up to track all upstream branches for the boards, then just fetch from the master branch locally? I am trying to create a patch based on the Collabora WIP patches for V4L2 HEVC work, and dealing with the build script re-fetching the mainline kernel from the master repository is just too much. So I have cloned the tree myself into my home dir, and changed lib/configuration.sh MAINLINE_KERNEL_SOURCE to point at the $HOME/linux-source/.git directory. Armbian then takes over and does the full clone locally, and I can tweak/rebase my local copy as needed, without needing to change OFFLINE_WORK and dealing with refetching different current/edge versions.

going · June 11, 2022

@hege In a sense, all the participants of the armbian project can be called free artists. In this sense, the saying can be applied to all of us: "he is an artist and he sees this way." I implemented the "OFFLINE_WORK" key a long time ago. The main purpose of this work was to ensure that the assembly system did not make requests to distant countries and did not wait for long-term responses. This greatly slowed down the assembly process.
And also do not update local repositories, but only return them to their original state.
Today, on my local host, I implemented support for the work of the git daemon. And all requests from the VM are forwarded there. The relevance of this key has become minimal. But it remained in the build system and I hope that it will be useful to someone.