- stop at the correct places for `inventory` and `targets` non-GHA-specific commands
- handle `gha-template` first (and return), since it does not really depend on any info/targets/etc
- batcat the output for show-off
- python-tools: add Jinja2. Incredible how we made it this far without it.
- output-gha-workflow-template: a double-templater, first runs jinja with a custom syntax, then "moar magic" to be useful for GHA
* Move packages to new location
* Adjust build runners
* Switch to OS
Signed-off-by: Igor <igor@armbian.com>
---------
Signed-off-by: Igor <igor@armbian.com>
Co-authored-by: Ricardo Pardini <ricardo@pardini.net>
- rootfs is not a deb package, and thus not subject to the same strict ordering rules of `artifact_prefix_version`
- replace with `yyyymm` (`202305`) in version so we get one-per-month automatically
- all interactive commands now **don't build the artifact** anymore; just patches/.configs are produced and then build ends
- user is required to put the produced patches in the right place and build again, for full consistency
- split ATF and U-BOOT manual patching process; use CLI command `atf-patch` to patch ATF, and `uboot-patch` to patch u-boot
- non-interactive artifact builds are now 100% sans-stdin
- introduce `uboot-config` CLI command; still experimental, only produces a defconfig and not a patch
- reworked `userpatch_create()` to be (hopefully) more useful:
- detects a previous patch and offers to apply it before continuing
- enters a loop showing the diff, and only proceeds when user indicates he's happy with the patch
- produces `mbox`-formatted patches via `format-patch` and standard Armbian parameters
- uses MAINTAINER and MAINTAINERMAIL instead of git configuration (so it works in containers)
- don't allow image builds with any patching or configuring _at all_ (it has been deprecated with a warning for months already, and results are inconsistent)
- determinining if `/boot` is vfat is more complex than it looks;
- use a common function to unify across all scripts
- during image build, the kernel install is always done with a non-fat, non-mounted /boot
- use a special variable passed through the env so unified function can know ahead of time
- introduce `extra_apt_envs` param to `chroot_sdcard_apt_get()`; array with extra environment vars
- if vfat /boot:
- `linux-update-symlinks` is not called (Debian would just choke on this)
- don't even try to symlink Armbian stuff, move instead
- more/better logging board-side (unfortunately mixed in all the set -x debugs)
- pipeline: add `pipeline` context object to targets; use it to filter artifacts and images to build; warn about oci-name with multiple oci-tags
- pipeline: better error messages when info's fail; show some (out-of-order) progress messages during parallel info gathering
- pipeline: targets-compositor: add `not-eos` inventory
- TARGETS_FILENAME, log all OCI lookups
- SKIP_IMAGES
- IMAGES_ONLY_OUTDATED_ARTIFACTS
- no dash in chunk id in JSON
- pipeline: very initial chunking, using the same outputs
- pipeline: template targets, `items-from-inventory:` inventory expansion, CHECK_OCI=yes, CLEAN_MATRIX=yes, CLEAN_INFO=yes, many fixes
- cli: `inventory` / `targets` / `matrix` / `workflow`
- pipeline: workflow beginnings
- pipeline: general log cleanup + OCI stats / better miss handling
- pipeline: fixes/reorg
- pipeline: catch & log JSON parsing errors
- pipeline: gha matrix: use IMAGE_FILE_ID as job description
- pipeline (delusion): gha workflow output, based on old matrix code
- pipeline: better parsing and reporting of stderr log lines (under `ANSI_COLOR=none`)
- pipeline: mapper-oci-uptodate: use separate positive/negative cache dirs (GHA will only cache positives); cache negs for 5 minutes locally
- pipeline: output-gha-matrix artifacts + images
- pipeline: output-gha-matrix artifacts + images: "really" and fake 1-item matrix if empty
- pipeline: move files into subdir; update copyright & cleanup
- pipeline: refactor bash jsoninfo driver a bit
- pipeline: outdated-artifact-image-reducer
- pipeline: introduce `target_id` at the compositor, aggregate it at the reducer, carry it over in the artifact info mapper
- pipeline: mapper-oci-uptodate
- pipeline: info-gatherer-artifact, with PRE_PREPARED_HOST
- pipeline: refactor/rename info-gatherer-image.py
- pipeline: beginnings
- git-ref2info: when resolving SHA1's for 'xxx' tag or branch, try first `refs/heads/xxx`, then just 'xxx'; validate SHA1 with 40-char regexp
- this works-around Gerrit-style repositories that have a 'refs/for/xxx' ref that gets in the way
- case in point: https://github.com/hardkernel/u-boot/tree/odroidc-v2011.03 (has both refs/heads/odroidc-v2011.03 and refs/for/odroidc-v2011.03)
- git-ref2info: when processing tags for a SHA1, try both dereference and non-dereferenced, in that order, to get a SHA1
- this solves old problem reported in #4916: GitHub incorrectly processes dereference requests for tags that are not actually annotated
- case in point: https://github.com/ayufan-rock64/linux-kernel/releases/tag/4.4.202-1237-rockchip-ayufan
- `codeaurora` is no more, https://bye.codeaurora.org/
- allow adding re-launch environments by populating `ARMBIAN_CLI_RELAUNCH_PARAMS` dict
- similar to `ARMBIAN_CLI_RELAUNCH_PARAMS`
- rename/consolidate `()`'s output globals:
- `ARMBIAN_CLI_FINAL_RELAUNCH_ARGS` (rename)
- `ARMBIAN_CLI_FINAL_RELAUNCH_ENVS` (new)
- includes `ARMBIAN_HIDE_REPEAT_PARAMS` automatically
- `sudo`: include envs when relaunching, use bash explicitly
- `docker`: better logging
- `docker`: include envs when relaunching via `--env` args
Add interactive configurations to Repeat Build Options
Simplify function produce_repeat_args_array()
- make use of $ARMBIAN_NON_PARAM_ARGS and $ARMBIAN_PARSED_CMDLINE_PARAMS (associated array)
- quote parameter values to be on the safe side
Move "Repeat Build Options" logs to start-end.sh
- cli-build.sh, start-end.sh:
- move function function produce_repeat_args_array() to start-end.sh
- add $WHAT as first arg to produce_repeat_args_array() if
$WHAT is not empty
- move early display log of "Repeat Build Options" to
function main_default_start_build()
- move last log of "Repeat Build Options" to
function main_default_end_build()
- add last log of "Repeat Build Options" to
LOG_SECTION="repeat_build_options"
- rootfs: rootfs-create: show a summary of the 20 biggest dirs, right before tarring the rootfs (for debugging)
- rootfs: rootfs-create: show usage of caches between first and second stages
- rootfs: rootfs-create: cleanup junk left by `debootstrap` after second stage
- rootfs: rootfs-create: _always_ clean apt stuff at the end
- rename `apt_purge_unneeded_packages()` to `apt_purge_unneeded_packages_and_clean_apt_caches()` for clarity
- image: `apt_purge_unneeded_packages_and_clean_apt_caches()`: warn if apt caches not empty; clean them off, always.
- host-utils: `local_apt_deb_cache_prepare()`: also test the target, warn if not empty
- extension: cleanup-space-final-image: do NOT clean apt stuff. done in core now
- the metric shit-ton of debugs added should help the next person who faces this in the future
- can add to `EXTRA_IMAGE_SUFFIXES` array during `user_config` or `extension_prepare_config`
- in the end goes into a `EXTRA_IMAGE_SUFFIX` global readonly
- this simplifies extension code
- we still can't "predict" image names during configuration since it includes the kernel version (6.2.11/generic) which only comes later
- if CONFIG_DEFS_ONLY during `obtain_complete_artifact()`, just dump JSON to stdout after calculating version/coordinates
- don't prepare ORAS tooling if CONFIG_DEFS_ONLY
- don't run in standard build (and thus don't prepare_host()) if under CONFIG_DEFS_ONLY
- if CONFIG_DEFS_ONLY, don't run `check_basic_host()`
Addressing:
[🐳|💥] Error context msg [ Installation of Armbian main packages for sid 3dsupport xfce no failed ]
Error: Error context msg Installation of Armbian main packages for sid 3dsupport xfce no failed
[🐳|🔨] Reading package lists...
[🐳|🔨] Building dependency tree...
[🐳|🔨] E: Unable to locate package firmware-sof-signed
- fix: indent of generated sources
- fix: no reason to source the generated file in a loop. do it once. 20x faster?
- `dump_extension_method_sources_functions "name_of_hook"`: get function definitions (inside braces)
- `dump_extension_method_sources_body "name_of_hook"`: get the function bodies (sans-braces)
- bsp-cli: does not require aggregation
- bsp-desktop / desktop: requires aggregation
- 'desktop' packaged moved out of 'bsp' folder, it is NOT a bsp
- make consistent naming/folders
- extract .sh generation workflow into 'declare -f' magic functions
- `SHOW_DEBIAN=yes` to bump most of `DEBIAN/xxx` stuff to screen
- artifact armbian-bsp-desktop: use aggregation, require certain vars, CLI forces `BUILD_DESKTOP=yes`; requires BOARD
- artifact armbian-desktop: use aggregation, require certain vars, CLI forces `BUILD_DESKTOP=yes`, not require `BOARD` anymore
- cli: commands: fix artifact aliases, reorder
- desktop/bsp-desktop: add `config_dump` method; fix `artifact_map_debs` ref subdir and name
- artifact bsps: require aggregation to build, not to list what needs to be built
- distro-agnostic: fix install of `armbian-bsp-cli` (not `armbian-bsp`)
- armbian-bsp-cli: add `_config_dump` method, use configuration with aggregation, add required vars check
- distro-specific: fix installation of `fake-ubuntu-advantage-tools`
- include rudimentary artifact dependency information for config-dump-json: fixes/refactor/cleanup
- artifact-rootfs.sh: include rudimentary artifact dependency information for config-dump-json
- artifact-armbian-(config/plymouth-theme/zsh).sh: include rudimentary artifact dependency information for config-dump-json
- artifact-fake-ubuntu-advantage-tools.sh: include rudimentary artifact dependency information for config-dump-json
- use new capture'd vars scheme
- so `./compile.sh BOARD=xxx BRANCH=yyyy config-dump-json | jq .` now works and is consistent/newline tolerant
- introduce internal `skip_host_config=yes` for `prep_conf_main_minimal_ni()` to skip calling `check_basic_host()`
- doesn't show nor clear vars `if_error_find_files_sdcard` and `if_error_detail_message`
- used by `chroot_sdcard_apt_get()` which uses `run_host_command_logged()` multiple times, and the first would clear the important/last one
- giving up on the "bright black" == "gray" thing. There's enough bad default themes out there, and not the first time people (rightfully) complained...
Changing export-logs.sh script from posting to armbian.next.armbian.com to posting to the primary paste server which has now been upgraded at paste.armbian.com.
* Add / modify (c) in bash scripts
Signed-off-by: Igor <igor@armbian.com>
* Add (c) to the source config files
---------
Signed-off-by: Igor <igor@armbian.com>
* main source for driver and patches are sunxi64 megous patches
* move into patch/misc/wireless-rtl8723cs directory
* integrate with minor patches from rk322x/rockchip64 families
* update drivers_network.sh to apply patches in same order as before
* remove references from patches.megous and series.conf in sunxi64
* remove patches from rk322x and rockchip64 family patches
* do not touch kernel archives older than 6.1
- bump version to 23.05
- adjust OCI repositories, changed hash lenght to 12 and removed rootfs- prefix to shorten names (costmetic adjustement)
- build configs were moved to https://github.com/armbian/os
- remove CI that is not going to work anymore. This means also master is out of this CI
- small changed to Readme.txt. Just adding some note, removing obvious
- add propery - supported arhitectures for distribution. Needed for CI optimisation
- remove README.armbian-next.md (need to be somewhere else)
- shrink CODEOWNERS due to messages overload
Signed-off-by: Igor <igor@armbian.com>
- compress *all* present output images, not only .img
- remove 7zip compression & hostdep
- squash a few shortcircuits (yeah, this late in the game...)
- artifacts: introduce `ARTIFACT_IGNORE_CACHE=yes`
- artifacts: introduce `DONT_BUILD_ARTIFACTS`, list of artifacts that if not found cached, fail the build
- kernel_package_source() is no more
- a long dissertation about kernels, families, and the universe
- artifacts: actually use rootfs artifact for image build
- artifacts: detangle via artifact_base_dir
- artifacts: rootfs: use folders in artifact_name; include cache_type
- artifacts: some cleanups / validations
- rootfs artifact; drop old rootfs cli
- artifacts: new CLI shortcuts; remove old firmware CLI
- artifacts: full firmware & usage
- use firmware artifacts in image build and install its debs
- kernel artifact sans legacy; use tmpdir for .deb target for all packages
- legacy artifact versions is no more; pack/unpack now in common obtain;
- artifacts: uboot: cleanup legacy renaming, use artifact version directly
- artifacts: add firmware (small) artifact
- deploy uboot to loop from artifact; allow tty to artifact; todos for cleaning
- fixes, kernel dtb/headers conditional; remove `.git` from Makefile url; use mapfile for finding files to hash
- completely remove KERNEL_HAS_WORKING_HEADERS_FULL_SOURCE and `kernel_package_callback_linux_headers_full_source()`
- don't use realpath for artifact_file_relative
- curb some warnings
- fix: only install headers & dtbs if such exist
- kernel .config hook modification hash versioning
- OCI_TARGET_BASE vs per-artifact defaults; only deploy to remote from CLI with OTB
- artifact revolver & installing into image
- add artifact_map_packages and artifact_map_debs dicts
- revolver accumulates all info
- REPOSITORY_INSTALL is no more (for uboot/kernel, later others)
- rename `VER` to `IMAGE_INSTALLED_KERNEL_VERSION`
- Remove PGP signing from the script as its done in GHA
- Also remove generating .current control file as its obsolete in this design
Co-authored-by: Ricardo Pardini <ricardo@pardini.net>
- rootfs: create readonly global `rootfs_cache_id` (type+hash+date/version)
- add "oras-upload" CLI command (dumb, no retries, single target/single source)
- docker: mount-bind for & re-pass envs `GITHUB_OUTPUT` and `GITHUB_STEP_SUMMARY` down to Docker
- introduce `github-actions.sh::github_actions_add_output()`
- during logs cleanup, dump the Markdown log into GITHUB_STEP_SUMMARY if it is available
- killed `[[ $ROOTFS_TYPE != ext4 ]] && display_alert "Assuming ${BOARD} ${BRANCH} kernel supports ${ROOTFS_TYPE}" "" "wrn"`
- which definitely didn't belong in rootfs
- disable usage of run_host_command_logged_long_running for kernel-make
- stop lying about long_running stuff being any different from non-long_running versions
- stop lying about 'set -e' when there's still a bunch of pipes for "pv" and stuff all around
- require ARCH/RELEASE for rootfs cli
- disallow BOARD/LINUXFAMILY/BOARDFAMILY being set for rootfs cli
- split `prep_conf_main_only_rootfs()` instead of `prep_conf_main_build_single()` for rootfs-only stuff
- run both conf and build without stdin for rootfs CLI (no interactive allowed)
- don't blindly set `ARCH=armhf` for no reason in main-config
- allow skipping family stuff in main-config, via `allow_no_family=yes` (used only in `prep_conf_main_only_rootfs()`)
- kill double loading of userpatches family, not needed/old code
- allow skipping kernel stuff via `skip_kernel=yes` in `config_pre_main()` (used only in `prep_conf_main_only_rootfs()`)
- make `tmpfs_estimated_size` and `available_physical_memory_mib` readonly and global and only ever set once
- remove subshell from `get_rootfs_cache_list()`, new version `get_rootfs_cache_list_into_array_variable()` for sanity
- warns if `ROOT_FS_CREATE_ONLY=yes` is set during cached build path
- add a `wait_for_disk_sync()` after restoring rootfs cache
- fix logging sections for `build_rootfs_only()` and `prepare_rootfs_build_params_and_trap()` in `build_rootfs_and_image()`
- remove the logging section from `get_or_create_rootfs_cache_chroot_sdcard()` call in `build_rootfs_only()` (which is now itself logged)
- @TODO: `deploy_qemu_binary_to_chroot()` is leaking a qemu bin in rootfs cache!
- _told ya it would all make sense eventually..._
- introduce new, real, non-alias `rootfs` command in `cli-rootfs` -- this _only_ builds rootfs, nothing else
- no more `main_default_build_single()`; instead `full_build_packages_rootfs_and_image()`
- introduce `do_with_default_build()` wrapper, for things that need host prepared + aggregation to run
- CLI-specific parts of `main_default_build_single()` moved to cli-build
- put prepare/cleanup pair of `prepare_rootfs_build_params_and_trap()` and `trap_handler_cleanup_rootfs_and_image()` into new `trap-rootfs.sh` (this needs further splitting for the image part)
- refactor `get_or_create_rootfs_cache_chroot_sdcard()` into `build_rootfs_only()`, `calculate_rootfs_cache_id()`
- update library with new files
- "| cut" masked errors in multiple places
- include real pip3 version in python deps hash, all will be invalidated
- be less happy about hitting hash/cache, should happen often
- allows using pre-shallow-ed bare trees, specific to the KERNEL_MAJOR_MINOR
- uses shallow by default if storage is "slow" (mmc), or free disk space < 32g (@TODO tweak this)
- if full tree already there keeps using it
- allow forcing decision with `KERNEL_GIT=full` or `KERNEL_GIT=shallow`
- countdown to confirm if decision based on storage type/free space and user is interactive
- this has some of the worst copy ever written
- @TODO refactor the device type / free space code, reuse across codebase (3-4 other spots detect free space)
- @TODO GHA self-hosted (full) vs hosted (shallow) runners decision
- sprinkle with `wait_for_disk_sync()` around clean/download/extract
- sprinkle git with debugs, and an info between fetch and checkout (otherwise it seems "fetch" takes a long time)
- wait 10s for sync call; otherwise warn user to be patient
- wait_for_sync "reason for sync" everywhere
- add @TODO for /run/user/0 etc and XDG_RUNTIME_DIR in chroot
- rpardini: actually use ports keyring for Debian Risc-V
- `risc-v`: correctly include `debian-ports-archive-keyring` in Risc-V CLI packages
- include in debootstrap list, via `config/optional/architectures/riscv64/_config/cli/_all_distributions/main/packages`
- can't have two `--include` in debootstrap call
- riscv64: move `debian-ports-archive-keyring` from main to debootstrap; also only for `sid`
- don't show stats for patches that are bind applied blind (failed to parse/autogen)
- don't show Author, it's too long
- don't dump Python exceptions in log
- extra: expand Python imports
- reorg `display_alert()` so more frequent levels are handled first
- improve some ANSI color usage, still a mess
- don't write failed commands stackstrace as warnings to screen -- just to file (yes it's redundant, except if in test)
- more useful / subdued stacktraces, to make actual error more obvious...
