RSS Bot Posted August 11, 2023 Posted August 11, 2023 hashed-OCI-revisioned-debs: introduce "reversioning" of .deb packages trap-logging: just discard logs dir when ANSI_COLOR=none repo-reprepro.py: set -e & pipefail prepare_host: handle PRE_PREPARED_HOST inside prepare_host_init() so WORKDIR etc is always available kernel-debs: avoid showing tree of kernel modules if they've not been built for example EXT=nomod causes no modules to be built debs: all produced debs now get a placeholder DEBIAN/changelog and a usr/share/doc .gz changelog with hash artifacts: don't keep deb-tar's .tar after uploading to OCI they won't ever be used after this, and just accumulate trash for no reason hashed-OCI-revisioned-debs: get completely rid of artifact_prefix_version hashed-OCI-revisioned-debs: build debs in PACKAGES_HASHED_STORAGE, not DEB_STORAGE (temp commit, will be rewritten by a later commit) fakeroot_dpkg_deb_build() now only takes a single argument, the unpacked package dir kernel/tmpfs: remove usage of the kernel-specific tmpfs for temporary kernel .deb's, now build directly to packages-hashed dir uboot: drop CHOSEN_UBOOT, change deployment directory, remove uboot_name as temp_dir prefix change deployment dir to not include REVISION or ARCH get rid uboot's CHOSEN_UBOOT, REVISION and ARCH in the directory name. no two u-boot debs can be installed in the same machine anyway pipeline: output-debs-to-repo-json.py: adapt to new reversioned JSON info hashed-OCI-revisioned-debs: introduce "reversioning" of .deb packages tl-dr: maximize OCI cache hit ratio across nightlies/releases/PRs/etc; publish simple Version:'s that don't include a crazy hash in repo and images introduce output/packages-hashed directory radically change the output/debs directory structure simplify artifact's prepare_version() method for deb and deb-tar artifacts: artifact_base_dir and artifact_final_file will now be auto-calculated; thus removed from each artifact (except rootfs) artifact_deb_repo ("global", "jammy", "bookworm") is now required; "global" means common across all RELEASES artifact_deb_arch is now required, "all" is arch-independent, otherwise use ${ARCH} artifact_map_debs is now auto-calculated based on the above, and shouldn't be specified manually artifact_final_version_reversioned is optional, and can force the final version of the artifact (specific for the base-files case) artifacts that need special handling for reversioning can add function names to artifact_debs_reversion_functions array (base-files and bsp-cli cases) artifacts prepare_version() should set artifact_version, but never include it in other variables; artifact_version is now changed by framework after prepare_version() returns no longer use/refer/mention ${REVISION} when building packages. All packages should be ${REVISION}-agnostic. ${REVISION} (actually, artifact_final_version_reversioned) will be automatically swapped in the control file during reversioning fakeroot_dpkg_deb_build() now takes exactly two arguments: the directory to pack, and the deb ID (key of artifact_map_packages dict); add this change in all the artifact's code for this obtain_complete_artifact(): automatically adds -Rxxxx "revisioning-hash" to artifact_version, by hashing the revisioning functions and any artifact_debs_reversion_functions set calculates more complex subdirectory paths for both the output/packages-hashed and output/debs/output/debs-beta directories with the new subdirectories we can be sure a re-version is already done correctly and can skip it (eg, for partial download-debs re-runs) in the future we can automatically clean/remove old versions that are no longer relevant based on the dir structure exports a lot more information to JSON, including the new subdirectory paths comment-out code that implemented skip_unpack_if_found_in_caches, I'm very unsure why we had this in the first place obtain_artifact_from_remote_cache() for deb type artifacts, OCI won't preserve the subdirectory structure, so move downloaded files to the correct subdirectory manually this is not needed for deb-tar, since that can preserve the dir structure itself introduce artifacts-reversion.sh and its main function artifact_reversion_for_deployment() this has the logic for reversioning .deb's, by ar-unpacking them, changing control.tar (and possibly data.tar), handling .xz compression, etc. also handles hashing those functions, for consistency. Any changes in reversioning code actually change the artifact itself so we're not caught by surprise by default, it changes control file only: replace Version: (which is the hash-version originally) with artifact_final_version_reversioned (which is mostly just ${REVISION}) add a custom field Armbian-Original-Hash: with the original hash-version artifact_reversion_for_deployment() is called by new CLI wrapper cli_obtain_complete_artifact(), used for CLI building of specific artifact, but also for download-artifact build_artifact_for_image() used during image build armbian-bsp-cli-deb.sh: move ${REVISION} related stuff from the main package build to new reversioning functions. artifact-armbian-base-files.sh: move ${REVISION} related stuff from the main package build to new reversioning functions. kernel: add some custom fields to DEBIAN/control: Armbian-Kernel-Version: / Armbian-Kernel-Version-Family: (for future use: cleanup of usage of Source: field which should be removed) declutter the Description: field, moving long description out of the first line obtain IMAGE_INSTALLED_KERNEL_VERSION from the reversioned deb (this is still a hack and has not been fixed) uboot: declutter the Description: field, moving long description out of the first line use the reversioned .deb when deploying u-boot to the image main_default_build_packages() now stores reversioned values and complete paths to reversioned .deb's list_installed_packages() now compares custom field Armbian-Original-Hash: , and not the Version: to make sure debs in the image are the ones we want install_artifact_deb_chroot() is a new wrapper around install_deb_chroot() for easy handling of reversioned debs use it everywhere install_deb_chroot() was used in distro-agnostic.sh and distro-specific.sh main-config: allow using REVISION from env or command-line param; ensure it begins with digit pipeline: force compositor to include REVISION= just like it already did for BETA= this way the prepare step's REVISION (possibly passed via CLI parameter) is always sovereign pipeline: streamline output-debs-to-repo-json to pass down info and avoid parsing paths completely artifacts: obtain: back to using flat output/debs structure, include revisioned and hashed in filename, but no subdirectories handle the global artifact_deb_repo case specially (they go in the root, not subdir) adapt obtain_complete_artifact() and friends to new reversioned scheme; introduce UPLOAD_TO_OCI_ONLY=yes for deploying to remote cache tl-dr: only deploys to remote OCI if UPLOAD_TO_OCI_ONLY=yes; stop leaving junk behind in local cache in many situations simplify CLI artifact building parameters and behaviour ARTIFACT_USE_CACHE is now deprecated, and its behaviour is the default for any uploading to OCI to occur, UPLOAD_TO_OCI_ONLY=yes must be present; in this case, reversioning is not done FORCE_ARTIFACTS_DOWNLOAD is completely removed (use download-artifact instead) cli_obtain_complete_artifact()'s and build_artifact_for_image()'s reversioning is now moved to common obtain_complete_artifact() standard_artifact_reversion_for_deployment(): check for hashed deb existence only if reversioned does not exist touch the reversioned file if it already exists; helps to clean up junk later delete hashed version after reversioning, so we don't leave trash behind unless in download-artifact mode, which touches the hashed version we can later delete old files from packages-hashed to keep junk under control refactor obtain_complete_artifact() extract function artifact_dump_json_info() since obtain is large enough already when deploying to remote, always ignore all local and remote caches introduce artifact_is_available_in_revisioned_local_cache() if not deploying to remote, and revisioned cache exists, use it directly if deploying to remote, reversioned is not checked and not created if deploying to remote, force DEB_COMPRESS=xz if deploying to remote, completely remove the local cache base dir after upload is done (no more junk leftover) store dict of artifacts actually installed in image; use it to freeze (BSPFREEZE=yes) and check hashes without spurious errors eg: linux-headers might, or might not, be installed; same with bsp-cli transitional accidentally fixes AR-1802 start-end: undo unintentional damage done to python/python2 compatibility symlinks inside the BIN_WORK_DIR kernel: fix: when KERNEL_SKIP_MAKEFILE_VERSION=yes, use 1- prefix for artifact_version (eg odroidxu4) artifact-{bsp-cli,kernel,uboot}: fix missing vars_config_hash "variables hash" in artifact_version_reason View the full article
Recommended Posts