fdc1966847
32 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Ricardo Pardini
|
2142f73b97 |
hashed-OCI-revisioned-debs: introduce "reversioning" of .deb packages
> tl-dr:
> - maximize OCI cache hit ratio across nightlies/releases/PRs/etc;
> - publish simple `Version:`'s that don't include a crazy hash in repo and images
> - introduce `output/packages-hashed` directory
> - radically change the `output/debs` directory structure
- simplify artifact's `prepare_version()` method for `deb` and `deb-tar` artifacts:
- `artifact_base_dir` and `artifact_final_file` will now be auto-calculated; thus removed from each artifact (except `rootfs`)
- `artifact_deb_repo` ("global", "jammy", "bookworm") is now required; "global" means common across all RELEASES
- `artifact_deb_arch` is now required, "all" is arch-independent, otherwise use `${ARCH}`
- `artifact_map_debs` is now auto-calculated based on the above, and shouldn't be specified manually
- `artifact_final_version_reversioned` is optional, and can force the final version of the artifact (specific for the `base-files` case)
- artifacts that need special handling for reversioning can add function names to `artifact_debs_reversion_functions` array (`base-files` and `bsp-cli` cases)
- artifacts `prepare_version()` should set `artifact_version`, but _never_ include it in other variables; `artifact_version` is now changed by framework after `prepare_version()` returns
- no longer use/refer/mention `${REVISION}` when building packages. All packages should be `${REVISION}`-agnostic.
- `${REVISION}` (actually, `artifact_final_version_reversioned`) will be automatically swapped in the `control` file during reversioning
- `fakeroot_dpkg_deb_build()` now takes exactly two arguments: the directory to pack, and the deb ID (key of `artifact_map_packages` dict); add this change in all the artifact's code for this
- `obtain_complete_artifact()`:
- automatically adds `-Rxxxx` "revisioning-hash" to `artifact_version`, by hashing the revisioning functions and any `artifact_debs_reversion_functions` set
- calculates more complex subdirectory paths for both the `output/packages-hashed` and `output/debs`/`output/debs-beta` directories
- with the new subdirectories we can be sure a re-version is already done correctly and can skip it (eg, for partial `download-debs` re-runs)
- in the future we can automatically clean/remove old versions that are no longer relevant based on the dir structure
- exports a lot more information to JSON, including the new subdirectory paths
- comment-out code that implemented `skip_unpack_if_found_in_caches`, I'm very unsure why we had this in the first place
- `obtain_artifact_from_remote_cache()`
- for `deb` type artifacts, OCI won't preserve the subdirectory structure, so move downloaded files to the correct subdirectory manually
- this is not needed for `deb-tar`, since that can preserve the dir structure itself
- introduce `artifacts-reversion.sh` and its main function `artifact_reversion_for_deployment()`
- this has the logic for reversioning .deb's, by `ar`-unpacking them, changing `control.tar` (and possibly `data.tar`), handling `.xz` compression, etc.
- also handles hashing those functions, for consistency. Any changes in reversioning code actually change the artifact itself so we're not caught by surprise
- by default, it changes `control` file only:
- replace `Version:` (which is the hash-version originally) with `artifact_final_version_reversioned` (which is mostly just `${REVISION}`)
- add a custom field `Armbian-Original-Hash:` with the original hash-version
- `artifact_reversion_for_deployment()` is called by
- new CLI wrapper `cli_obtain_complete_artifact()`, used for CLI building of specific artifact, but also for `download-artifact`
- `build_artifact_for_image()` used during image build
- `armbian-bsp-cli-deb.sh`: move `${REVISION}` related stuff from the main package build to new reversioning functions.
- `artifact-armbian-base-files.sh`: move `${REVISION}` related stuff from the main package build to new reversioning functions.
- `kernel`:
- add some custom fields to `DEBIAN/control`:
- `Armbian-Kernel-Version:` / `Armbian-Kernel-Version-Family:` (for future use: cleanup of usage of `Source: ` field which should be removed)
- declutter the `Description:` field, moving long description out of the first line
- obtain `IMAGE_INSTALLED_KERNEL_VERSION` from the reversioned deb (this is still a hack and has not been fixed)
- `uboot`:
- declutter the `Description:` field, moving long description out of the first line
- use the reversioned .deb when deploying u-boot to the image
- `main_default_build_packages()` now stores reversioned values and complete paths to reversioned .deb's
- `list_installed_packages()` now compares custom field `Armbian-Original-Hash: `, and not the `Version:` to make sure debs in the image are the ones we want
- `install_artifact_deb_chroot()` is a new wrapper around `install_deb_chroot()` for easy handling of reversioned debs
- use it everywhere `install_deb_chroot()` was used in `distro-agnostic.sh` and `distro-specific.sh`
|
||
|
Ricardo Pardini
|
0a5740040e |
rootfs-to-image: find, warn, and remove dangling symlinks in FAT32 /boot
|
||
Igor Velkov
|
9b68c4e42a |
Add support for nilfs2 fs based image
Signed-off-by: Igor Velkov <iav@iav.lv> |
||
|
Ricardo Pardini
|
c692676da7 |
kernel-debs: postinst/etc: fix: unify is_boot_dev_vfat() function; make work during image build too via env var
- determinining if `/boot` is vfat is more complex than it looks; - use a common function to unify across all scripts - during image build, the kernel install is always done with a non-fat, non-mounted /boot - use a special variable passed through the env so unified function can know ahead of time - introduce `extra_apt_envs` param to `chroot_sdcard_apt_get()`; array with extra environment vars - if vfat /boot: - `linux-update-symlinks` is not called (Debian would just choke on this) - don't even try to symlink Armbian stuff, move instead - more/better logging board-side (unfortunately mixed in all the set -x debugs) |
||
|
Ricardo Pardini
|
7ea8e040d0 |
cli: flash: introduce flash CLI command; introduce hook post_build_image_write
|
||
|
Ricardo Pardini
|
d59271f903 |
config: refactor calculate_image_version() out of create_image_from_sdcard_rootfs(), make more flexible, predict IMAGE_FILE_ID during config
- default `IMAGE_INSTALLED_KERNEL_VERSION="generic"` instead of `"linux"` |
||
|
Ricardo Pardini
|
ac25ac18be |
config/extensions: consolidate EXTRA_IMAGE_SUFFIXES array as _the way_ to add to image filename, during configuration
- can add to `EXTRA_IMAGE_SUFFIXES` array during `user_config` or `extension_prepare_config` - in the end goes into a `EXTRA_IMAGE_SUFFIX` global readonly - this simplifies extension code - we still can't "predict" image names during configuration since it includes the kernel version (6.2.11/generic) which only comes later |
||
|
Ricardo Pardini
|
ed0b406265 |
lib: fix: replace undue export statements with declare -g
|
||
|
Ricardo Pardini
|
91b7b35fa3 |
rootfs-to-image/distro-agnostic: introduce IMAGE_VERSION, can be used to override VERSION file ($REVISION) just for image filename/issue/os-release
|
||
Igor Pečovnik
|
b7b8eb7b72
|
Add / modify (c) in bash scripts (#4922)
* Add / modify (c) in bash scripts Signed-off-by: Igor <igor@armbian.com> * Add (c) to the source config files --------- Signed-off-by: Igor <igor@armbian.com> |
||
Jianfeng Liu
|
fdcabaad3f
|
not rsync /boot when copying root partition (#4859) | ||
|
Ricardo Pardini
|
a2304f28b0
|
armbian-next: post_build_image: run hook first, then write to CARD_DEVICE, then compress, in that order
- compress *all* present output images, not only .img - remove 7zip compression & hostdep - squash a few shortcircuits (yeah, this late in the game...) |
||
|
Ricardo Pardini
|
9bffa5e749
|
armbian-next: artifacts: firmware/rootfs and actual usage of artifacts during image build
- artifacts: introduce `ARTIFACT_IGNORE_CACHE=yes` - artifacts: introduce `DONT_BUILD_ARTIFACTS`, list of artifacts that if not found cached, fail the build - kernel_package_source() is no more - a long dissertation about kernels, families, and the universe - artifacts: actually use rootfs artifact for image build - artifacts: detangle via artifact_base_dir - artifacts: rootfs: use folders in artifact_name; include cache_type - artifacts: some cleanups / validations - rootfs artifact; drop old rootfs cli - artifacts: new CLI shortcuts; remove old firmware CLI - artifacts: full firmware & usage - use firmware artifacts in image build and install its debs - kernel artifact sans legacy; use tmpdir for .deb target for all packages - legacy artifact versions is no more; pack/unpack now in common obtain; - artifacts: uboot: cleanup legacy renaming, use artifact version directly - artifacts: add firmware (small) artifact - deploy uboot to loop from artifact; allow tty to artifact; todos for cleaning - fixes, kernel dtb/headers conditional; remove `.git` from Makefile url; use mapfile for finding files to hash - completely remove KERNEL_HAS_WORKING_HEADERS_FULL_SOURCE and `kernel_package_callback_linux_headers_full_source()` - don't use realpath for artifact_file_relative - curb some warnings - fix: only install headers & dtbs if such exist - kernel .config hook modification hash versioning - OCI_TARGET_BASE vs per-artifact defaults; only deploy to remote from CLI with OTB - artifact revolver & installing into image - add artifact_map_packages and artifact_map_debs dicts - revolver accumulates all info - REPOSITORY_INSTALL is no more (for uboot/kernel, later others) - rename `VER` to `IMAGE_INSTALLED_KERNEL_VERSION` |
||
|
Ricardo Pardini
|
72c61f2f68
|
armbian-next: replace bare sync invocations by introducing wait_for_disk_sync(); sync early at start of build
- wait 10s for sync call; otherwise warn user to be patient - wait_for_sync "reason for sync" everywhere - add @TODO for /run/user/0 etc and XDG_RUNTIME_DIR in chroot |
||
|
Ricardo Pardini
|
35fbb7ed6c
|
armbian-next: small cleanups: squash typos / add function keyword / add types / mark dead code | ||
|
Ricardo Pardini
|
af7ace7e57
|
armbian-next: don't leak .raw image if failure before move; show rootfs-tmpfs info
- add escapings and TODO's around the area... |
||
|
Ricardo Pardini
|
1636a2afea
|
armbian-next: tune logging: unmounting, u-boot prep/source/install; traps; extensions | ||
|
Ricardo Pardini
|
ff4b5069ce
|
armbian-next: extract move_images_to_final_destination() and optimize for fast-move when src/dst on same filesystem
|
||
|
Ricardo Pardini
|
d6200a30e6
|
armbian-next: make superglobals readonly (DEST/WORKDIR/LOGDIR/SDCARD/MOUNT and others)
|
||
|
Ricardo Pardini
|
12ffcd6319
|
armbian-oleg: curb rsync logging during bsp
|
||
|
Ricardo Pardini
|
8aa0ee640f
|
armbian-next: rootfs: don't stop if df -h fails
- still needs work |
||
|
Ricardo Pardini
|
690ca4df7c
|
armbian-next: add (trap) cleanup handler to reset ./tmp and ./output to original pre-sudo UID before exiting, even with failure/abort
|
||
|
Ricardo Pardini
|
3bc8afd593
|
armbian-next: refactor losetup -d ... into free_loop_device_insistent() which retries 10 times
|
||
|
Ricardo Pardini
|
d24d3327a8
|
armbian-next: the great cli entrypoint (+docker) rewrite; introduce USE_LOCAL_APT_DEB_CACHE replacing apt-cacher-ng
- armbian-next: introduce `USE_LOCAL_APT_DEB_CACHE` (default `=yes`) as alternative/in addition to `apt-cacher-ng` (eg, in Docker)
- this uses `cache/aptcache/${RELEASE}-${ARCH}` (in the host) for
- apt cache, by bind-mounting it to `${SDCARD}/var/cache/apt` in the `chroot_sdcard_apt_get()` runner and its usages
- debootstrap, by passing it `--cache-dir`
- utility function to help understand what is happening to cache during usage
- apt itself mantains this cache, removing old packages when new ones are installed. apt does this _by default_
- introduce `DONT_MAINTAIN_APT_CACHE=yes` to skip out of automatic apt maintenance of apt cache, eg, during `remove`s
- don't do `apt clean` and such if using local cache, that would clean the cache, not the chroot
- clean up `install_deb_chroot()` a little, find an unrelated bug there
- WiP: the great cli entrypoint (+docker) rewrite, Phase 6: relaunching structure; re-pass ARMBIAN_BUILD_UUID; use ARMBIAN_COMMAND for log filename; fix for output/logs dir perms
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 4/x; better logging, check & force `DEST_LANG`
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 3/x; don't write to stderr in generated Dockerfile
- it's `drastic red` on non-buildx dockers
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 2/x, logging
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 1/x
- source configs in a logging section.
- Docker: silent, fast retries to make sure `docker system df` works
- shut-up `chown` (no `-v`) output related to `SET_OWNER_TO_UID`
- ask user to wait while `DESTIMG` is rsync'ed to `FINALDEST` -- it's potentially very slow
- use green apple for Mac logging, instead of red apple which might imply error...
- WiP: the great cli entrypoint (+docker) rewrite, Phase 4: run as non-root, maybe-with-Docker
- introduce `is_docker_ready_to_go()`; if it is, and we're not root, use Docker instead of sudo. <- GOOD IDEA? BAD IDEA? lol
- introduce `SET_OWNER_TO_UID` var to be passed to Docker/sudo so written files are owned by the launching user, not root.
- introduce `mkdir_recursive_and_set_uid_owner()` and `reset_uid_owner()` to reset owner based on `SET_OWNER_TO_UID`
- use it for userpatches files created, logs, and output files, including images and debs.
- @TODOs ref. `$SUDO_USER` which I think the old version of this?
- add a lot of @TODOs, ref being able to relaunch something that's not `build` inside Docker, also add/change params and configs and command.
- initially add `ARMBIAN_DOCKER_RELAUNCH_EXTRA_ARGS`
- WiP: the great cli entrypoint (+docker) rewrite, Phase 3: rpardini is demented, v3
- WiP: the great cli entrypoint (+docker) rewrite, Phase 2: rpardini is demented
- WiP: the great cli entrypoint (+docker) rewrite, Phase 1
- armbian-next: WiP: Docker: actually use the GHA-image as base; pull it every 24hs.
- using image in my private repo.
- this has significant speedup to "start building time" on the 1st run
- move some Linux specific stuff to its own if
- add comments and todo
- armbian-next: WiP: Docker, high-WiP, beginnings of Armbian mount dict, with linux/darwin preferences
- armbian-next: WiP: Docker, configure `BUILDKIT_COLORS`
- armbian-next: WiP: Docker, make docker image from Dockerfile more compact by flattening layers
- armbian-next: `logging`: add whale indicator if build running under Docker
- armbian-next: WiP: `docker`: working with `bookworm`, `sid`, and `jammy` on Darwin & Linux; works with `bullseye` on Linux only
- armbian-next: WiP: `docker`: force ARMBIAN_RUNNING_IN_CONTAINER both in Dockerfile and passed as `--env`; apt update and install in same layer; back to jammy
- armbian-next: introduce `armbian_is_running_in_container()` and `armbian_is_host_running_systemd()`, replacing `systemd-detect-virt` in multiple spots
- WiP: try with debian:bullseye -- can't detect docker at all
- armbian-next: WiP: 2nd stab at new Docker support; Darwin still works; Linux `docker.io` working
- gen .dockerignore together with Dockerfile
- split in funcs
- hacks for Linux and `/dev/loop` stuff, CONTAINER_COMPAT=yes
- mac still works, Linux stuff would break it but I if'fed
- armbian-next: the secrets of `CONTAINER_COMPAT` revealed; add size checking to check_loop_device() and avoid retry when `mknod`ing
- this fails for the right reasons now, causing retries, which are then retried and work ;-)
- this is related to building under Docker on Linux, using docker.io package (not docker-ce)
- armbian-next: remove `.dockerignore` and add it to `.gitignore`; it's going to be auto-generated
- armbian-next: `.dockerignore`: Docker context should only have minimal files and folders, to speed up Dockerfile build
- IMPORTANT: `.dockerignore` is going to be generated from now on: so this is the last commit with changes before removal
- armbian-next: WiP: initial stab at new Docker support; really run the passed cmdline; add Dockerfile to gitignore
- armbian-next: WiP: initial stab at new Docker support; generate Dockerfile; introduce REQUIREMENTS_DEFS_ONLY
- uses REQUIREMENTS_DEFS_ONLY
- works on Docker Desktop on Mac;
- linux TBA
- armbian-next: don't error out if `.git` not present; other small fixes
- armbian-next: general "work or at least don't misbehave when run on a very bare ubuntu:latest instance"
- can't assume things, for example:
- that `sudo` will be available; it might not, and might be already root, no reason to fail
- that `/etc/timezone` will exist
- that `systemd-detect-virt` will be available
- that `git` will be available
- that `locale-gen` will be available
|
||
|
Ricardo Pardini
|
f921c8cd74
|
armbian-next: let all unmounting of ${MOUNT} be done by umount_chroot_recursive
|
||
|
Ricardo Pardini
|
d85bf2a155
|
armbian-next: many fixes after v29 rebase; mostly involving umount
- remove grub's --verbose, it's really too verbose - trust in TMPDIR, do not use trap at all in compile_plymouth-theme-armbian - settings traps adhoc is forbidden in armbian-next as well - be more verbose in umount_chroot(), do not try to unmount tmpfs-mounted dir itself, only xx/tmp - try to be smarter about /tmp being mounted in rootfs-to-image -- in the end the recursive saves us - run post_debootstrap_tweaks when the SDCARD is still mounted, not after - don't try to download anything from any mirrors if SKIP_ARMBIAN_REPO==yes |
||
|
Ricardo Pardini
|
833a7d9ab6
|
armbian-next: fix "create list of installed packages for debug" | ||
|
Ricardo Pardini
|
b25b3cf499
|
armbian-next: lib changes - MEGASQUASH - squashed changes from c9cf3fc241cfb4c872f4aef7bbc41d5854db7ea3 to 6809de3d6063cb041205a8318e19da6a4dee68c9 ref extensions_08_10_2022_pre_v30
- also compile.sh - shellfmt lib - split off shell and python tools under lib - revert removal of stuff a-n no longer uses (ref. compilation): general packaging, mkdeb etc - editoconfig split off - extension changes split off - sources and sources/families split off - some undue stuff removed or split armbian-next: manual merge (30) of lib changes between |
||
hzyitc
|
0b437bfd75
|
buildscript: Bugfix: handle over-mounted mountpoint (#4660)
Adjust the parameters when calculating the mount point |
||
Ash
|
408bc67619
|
Add shebangs for shellcheck (#4493)
* Add shebangs for shellcheck See #AR-1406 * Add shebangs for shellcheck Also for `extensions` scripts |
||
|
Ricardo Pardini
|
85c8cbf15d |
run shellfmt on all of lib
|
||
|
hzy
|
13eecd8240 |
Split lib/debootstrap.sh
|