RSS Bot Posted August 2, 2023 Posted August 2, 2023 hashed-OCI-revisioned-debs: introduce "reversioning" of .deb packages tl-dr: maximize OCI cache hit ratio across nightlies/releases/PRs/etc; publish simple Version:'s that don't include a crazy hash in repo and images introduce output/packages-hashed directory radically change the output/debs directory structure trap-logging: just discard logs dir when ANSI_COLOR=none repo-reprepro.py: set -e & pipefail prepare_host: handle PRE_PREPARED_HOST inside prepare_host_init() so WORKDIR etc is always available kernel-debs: avoid showing tree of kernel modules if they've not been built for example EXT=nomod causes no modules to be built debs: all produced debs now get a placeholder DEBIAN/changelog and a usr/share/doc .gz changelog with hash artifacts: don't keep deb-tar's .tar after uploading to OCI they won't ever be used after this, and just accumulate trash for no reason hashed-OCI-revisioned-debs: get completely rid of artifact_prefix_version hashed-OCI-revisioned-debs: build debs in PACKAGES_HASHED_STORAGE, not DEB_STORAGE (temp commit, will be rewritten by a later commit) fakeroot_dpkg_deb_build() now only takes a single argument, the unpacked package dir kernel/tmpfs: remove usage of the kernel-specific tmpfs for temporary kernel .deb's, now build directly to packages-hashed dir uboot: drop CHOSEN_UBOOT, change deployment directory, remove uboot_name as temp_dir prefix change deployment dir to not include REVISION or ARCH get rid uboot's CHOSEN_UBOOT, REVISION and ARCH in the directory name. no two u-boot debs can be installed in the same machine anyway pipeline: output-debs-to-repo-json.py: adapt to new reversioned JSON info hashed-OCI-revisioned-debs: introduce "reversioning" of .deb packages simplify artifact's prepare_version() method for deb and deb-tar artifacts: artifact_base_dir and artifact_final_file will now be auto-calculated; thus removed from each artifact (except rootfs) artifact_deb_repo ("global", "jammy", "bookworm") is now required; "global" means common across all RELEASES artifact_deb_arch is now required, "all" is arch-independent, otherwise use ${ARCH} artifact_map_debs is now auto-calculated based on the above, and shouldn't be specified manually artifact_final_version_reversioned is optional, and can force the final version of the artifact (specific for the base-files case) artifacts that need special handling for reversioning can add function names to artifact_debs_reversion_functions array (base-files and bsp-cli cases) artifacts prepare_version() should set artifact_version, but never include it in other variables; artifact_version is now changed by framework after prepare_version() returns no longer use/refer/mention ${REVISION} when building packages. All packages should be ${REVISION}-agnostic. ${REVISION} (actually, artifact_final_version_reversioned) will be automatically swapped in the control file during reversioning fakeroot_dpkg_deb_build() now takes exactly two arguments: the directory to pack, and the deb ID (key of artifact_map_packages dict); add this change in all the artifact's code for this obtain_complete_artifact(): automatically adds -Rxxxx "revisioning-hash" to artifact_version, by hashing the revisioning functions and any artifact_debs_reversion_functions set calculates more complex subdirectory paths for both the output/packages-hashed and output/debs/output/debs-beta directories with the new subdirectories we can be sure a re-version is already done correctly and can skip it (eg, for partial download-debs re-runs) in the future we can automatically clean/remove old versions that are no longer relevant based on the dir structure exports a lot more information to JSON, including the new subdirectory paths comment-out code that implemented skip_unpack_if_found_in_caches, I'm very unsure why we had this in the first place obtain_artifact_from_remote_cache() for deb type artifacts, OCI won't preserve the subdirectory structure, so move downloaded files to the correct subdirectory manually this is not needed for deb-tar, since that can preserve the dir structure itself introduce artifacts-reversion.sh and its main function artifact_reversion_for_deployment() this has the logic for reversioning .deb's, by ar-unpacking them, changing control.tar (and possibly data.tar), handling .xz compression, etc. also handles hashing those functions, for consistency. Any changes in reversioning code actually change the artifact itself so we're not caught by surprise by default, it changes control file only: replace Version: (which is the hash-version originally) with artifact_final_version_reversioned (which is mostly just ${REVISION}) add a custom field Armbian-Original-Hash: with the original hash-version artifact_reversion_for_deployment() is called by new CLI wrapper cli_obtain_complete_artifact(), used for CLI building of specific artifact, but also for download-artifact build_artifact_for_image() used during image build armbian-bsp-cli-deb.sh: move ${REVISION} related stuff from the main package build to new reversioning functions. artifact-armbian-base-files.sh: move ${REVISION} related stuff from the main package build to new reversioning functions. kernel: add some custom fields to DEBIAN/control: Armbian-Kernel-Version: / Armbian-Kernel-Version-Family: (for future use: cleanup of usage of Source: field which should be removed) declutter the Description: field, moving long description out of the first line obtain IMAGE_INSTALLED_KERNEL_VERSION from the reversioned deb (this is still a hack and has not been fixed) uboot: declutter the Description: field, moving long description out of the first line use the reversioned .deb when deploying u-boot to the image main_default_build_packages() now stores reversioned values and complete paths to reversioned .deb's list_installed_packages() now compares custom field Armbian-Original-Hash: , and not the Version: to make sure debs in the image are the ones we want install_artifact_deb_chroot() is a new wrapper around install_deb_chroot() for easy handling of reversioned debs use it everywhere install_deb_chroot() was used in distro-agnostic.sh and distro-specific.sh Signed-Off-By: Ricardo Pardini ricardo@pardini.net View the full article
Recommended Posts