Commit Graph

128 Commits

Author SHA1 Message Date
Ricardo Pardini
91094149de base-files / apt-utils: stop trying non "-updates" repos when looking for Ubuntu LTS release package
- otherwise, when the lookup at "jammy-updates" fails (due to server instability), and "jammy" works, we end up with the wrong version
- non-LTS releases don't have the "-updates" repo, so don't even try, which should make everything faster
- TODO: yet-another opportunity to have a release metadata file, otherwise this is yet-another place where we list "jammy"
2023-10-21 13:58:37 +02:00
Ricardo Pardini
1c5add5e51 don't break downloads if GITHUB_SOURCE is unset
- fixes f200587b86 from https://github.com/armbian/build/pull/5791
2023-10-10 20:25:53 +02:00
iysheng
f200587b86 Use GITHUB_SOURCE to accelerate download files from github 2023-10-08 16:45:00 +02:00
Ricardo Pardini
000f491617 kernel/memoize: allow customizing memoize's default TTL of 3600 seconds; allow customizing kernel TTL via KERNEL_GIT_CACHE_TTL 2023-09-29 14:25:58 +02:00
Andrew Davis
75bbc299fc extensions: Fix dump_extension_method_sources_functions() to iterate
The function dump_extension_method_sources_functions() dumps the content
of a single argument function. It is used in several spots were an array
is passed instead. We could fix all these call sites to iterate the call,
or we could change the called function to do the iteration. Do the latter.
2023-09-24 20:18:47 +02:00
amazingfate
1eaf25ad3d add mainline kernel mirror support 2023-08-23 21:34:48 +02:00
Ricardo Pardini
11df81403d config/lib/exts: run shellfmt
- `bash lib/tools/shellfmt.sh`
- mostly spacing changes that accumulated over time
2023-08-22 14:28:14 +02:00
Igor
2a8cb5793f Removing exceptions needed to support riscv64 withing Debian
Signed-off-by: Igor <igor@armbian.com>
2023-08-17 22:09:05 +02:00
Gunjan Gupta
bc0e131747 Enforce use of bind mount for /dev within chroot
The /dev/mapper directory created by devtmpfs lacked entry for
armbian-root thereby breaking the cryptsetup configuration generated
within initrd file. Use bind mount as that doesn't seem to suffer from
that issue.
2023-07-22 02:18:49 +05:30
Lane Jennison
ad97c1b7ee rockchip-rk3588 collabora: new branch (by @lanefu)
- `bsdextrautils` for kernel 6.4 (only if collabora branch)
- kernel 6.4 major, but skip rtw88 driver patching
2023-07-01 14:50:35 +02:00
Jens J
42de8b678c handle kernel.org forks 2023-06-26 13:45:04 +02:00
Andrew Davis
92aa7b3de2 Use HTTPS for kernel.org sources checkout
HTTPS is more secure and less likely to be blocked by firewalls. The
other sources here use HTTPS now, update the kernel to the same.
2023-06-14 15:55:09 +02:00
Ricardo Pardini
70a64a0415 patching: Patching Summary and Patching Failures tables using Python's Rich
- even Rich'er patch output by colorizing certain strings green/yellow/red
- BASE_GIT_TAG now very sneakily also accepts a branch name
- IMPORTANT: this includes: BREAKING CHANGE: patches failing to apply now break the build. fixes #4958
  - also break on legacy `process_patch_file()` failure, remove `EXIT_PATCHING_ERROR`
2023-06-13 12:29:52 +02:00
Ricardo Pardini
31131fe14e bsp-cli/BETA: fixes for bsp-cli bootscript hashing; don't normalize bootscript vars; debugging for BETA=yes/no stuff
also:
- get rid of comments and `SUBREVISION` and `RC` variables
- better debugging in calculate_hash_for_variables() do_normalize_src_path=yes/no
- tag a few places where output/debs might wrong in face of BETA=yes/no
2023-05-25 19:30:40 +02:00
Ricardo Pardini
15fcc0bb00 artifacts: introduce calculate_hash_for_variables() which normalizes the input; use in kernel/uboot/etc
- some hashed variables might contain "${SRC}", so hashes never match, unless built in Docker
  - this strips away SRC from all vars and adds debugging so we can detect more later.
- `artifact-uboot`: include more variables into hash, for ATF & rk stuff
- `artifact-armbian-desktop`:
  - hashed vars actually contain /armbian in different context, skip normalization in this case
  - include results of relevant aggregation into artifact_input_variables
    - otherwise: desktops with different appgroups/configs lead to build failures in pipeline
    - will cause warnings in JSON preparation step, if more than one appgroups/config combo is targeted, since repo can only have one
2023-05-24 19:23:08 +02:00
Ricardo Pardini
7d72d18bb0 git-ref2info: source.denx.de is a GitLab instance, add it to the selector; fixes #5168 2023-05-23 16:34:40 +02:00
Igor
e67215bd6a URL for riscv64 is different then expected
https://deb.debian.org/debian-ports/pool -> pool-riscv64
2023-05-21 11:11:07 +02:00
Ricardo Pardini
1fd5d519d4 introduce armbian-base-files artifact, which downloads & repacks base-files from upstream distro
- bsp-cli: now depends on `base-files (>= ${REVISION})`, this way upgrading the bsp-cli causes our base-files to be installed
  - bsp-cli no longer does gymnastics with /etc/os-release et al, all done in armbian-base-files now
- general/apt-utils.sh: introduce `apt_find_upstream_package_version_and_download_url()`
- base-files: add release to version, in order to comply with repo restrictions (valid repos can't have two different debs with same name and version, md5 must match)
2023-05-18 20:49:06 +02:00
Ricardo Pardini
e3a4060595 github-actions: more logging for GHA actions outputs 2023-05-16 20:09:47 +02:00
Ricardo Pardini
27a021d193 oci-oras: retry harder, and sleep for more time, if push failed 2023-05-16 20:09:47 +02:00
Ricardo Pardini
b456554610 artifacts(debs): rationalize "bash hash" creation with new calculate_hash_for_bash_deb_artifact() helper 2023-05-16 20:09:47 +02:00
Ricardo Pardini
78700cf139 logging/retries: curb CI/GHA warning emitted _for each retry_ 2023-05-15 17:43:25 +02:00
Ricardo Pardini
23a2b34847 pipeline: add output-gha-workflow-template.py utility, for rendering GHA workflow YAML templates with chunks
- python-tools: add Jinja2. Incredible how we made it this far without it.
- output-gha-workflow-template: a double-templater, first runs jinja with a custom syntax, then "moar magic" to be useful for GHA
2023-05-03 07:49:53 +02:00
Ricardo Pardini
f992dcd466 git-ref2info: retries for getting Makefile body (https://raw.githubusercontent.com has been failing a lot) 2023-05-01 22:47:03 +02:00
Ricardo Pardini
f8ddf7f9e2 🔥 JSON info pipeline: v18
- pipeline: add `pipeline` context object to targets; use it to filter artifacts and images to build; warn about oci-name with multiple oci-tags
- pipeline: better error messages when info's fail; show some (out-of-order) progress messages during parallel info gathering
- pipeline: targets-compositor: add `not-eos` inventory
- TARGETS_FILENAME, log all OCI lookups
- SKIP_IMAGES
- IMAGES_ONLY_OUTDATED_ARTIFACTS
- no dash in chunk id in JSON
- pipeline: very initial chunking, using the same outputs
- pipeline: template targets, `items-from-inventory:` inventory expansion, CHECK_OCI=yes, CLEAN_MATRIX=yes, CLEAN_INFO=yes, many fixes
- cli: `inventory` / `targets` / `matrix` / `workflow`
- pipeline: workflow beginnings
- pipeline: general log cleanup + OCI stats / better miss handling
- pipeline: fixes/reorg
- pipeline: catch & log JSON parsing errors
- pipeline: gha matrix: use IMAGE_FILE_ID as job description
- pipeline (delusion): gha workflow output, based on old matrix code
- pipeline: better parsing and reporting of stderr log lines (under `ANSI_COLOR=none`)
- pipeline: mapper-oci-uptodate: use separate positive/negative cache dirs (GHA will only cache positives); cache negs for 5 minutes locally
- pipeline: output-gha-matrix artifacts + images
  - pipeline: output-gha-matrix artifacts + images: "really" and fake 1-item matrix if empty
- pipeline: move files into subdir; update copyright & cleanup
- pipeline: refactor bash jsoninfo driver a bit
- pipeline: outdated-artifact-image-reducer
- pipeline: introduce `target_id` at the compositor, aggregate it at the reducer, carry it over in the artifact info mapper
- pipeline: mapper-oci-uptodate
- pipeline: info-gatherer-artifact, with PRE_PREPARED_HOST
- pipeline: refactor/rename info-gatherer-image.py
- pipeline: beginnings
2023-05-01 22:46:25 +02:00
Ricardo Pardini
59d6690a11 git-ref2info: many fixes for nonstandard repos; try a few different dereference and refs/ combinations in a certain order; remove codeaurora repo handling
- git-ref2info: when resolving SHA1's for 'xxx' tag or branch, try first `refs/heads/xxx`, then just 'xxx'; validate SHA1 with 40-char regexp
  - this works-around Gerrit-style repositories that have a 'refs/for/xxx' ref that gets in the way
  - case in point: https://github.com/hardkernel/u-boot/tree/odroidc-v2011.03 (has both refs/heads/odroidc-v2011.03 and refs/for/odroidc-v2011.03)
- git-ref2info: when processing tags for a SHA1, try both dereference and non-dereferenced, in that order, to get a SHA1
  - this solves old problem reported in #4916: GitHub incorrectly processes dereference requests for tags that are not actually annotated
  - case in point: https://github.com/ayufan-rock64/linux-kernel/releases/tag/4.4.202-1237-rockchip-ayufan
- `codeaurora` is no more, https://bye.codeaurora.org/
2023-04-28 21:32:52 +02:00
amazingfate
97a2e41f62 use common sed for GITHUB_SOURCE replace 2023-04-21 14:21:28 +02:00
Ricardo Pardini
8cd830d6e1 memoize-cached: add flock locking to run_memoized() to avoid error when run in parallel 2023-04-18 09:27:38 +02:00
amazingfate
a0f59bf914 use ghproxy to accelerate memoized_git_ref_to_info 2023-04-17 13:42:53 +02:00
Ricardo Pardini
9b229f25ad artifact-uboot: add hashes for extension and old-timey -Hooks and -Variables hash (BOOTDELAY et al)
- helper for hashing old-timey hooks: `calculate_hash_for_function_bodies()` in `hash-files.sh` cos I'm lazy
2023-04-03 21:21:03 +02:00
Ricardo Pardini
d2bb1f893a extensions: many fixes; allow for dumping the source of hooks, via dump_extension_method_sources_functions() and dump_extension_method_sources_body()
- fix: indent of generated sources
- fix: no reason to source the generated file in a loop. do it once. 20x faster?
- `dump_extension_method_sources_functions "name_of_hook"`: get function definitions (inside braces)
- `dump_extension_method_sources_body "name_of_hook"`: get the function bodies (sans-braces)
2023-03-31 09:12:44 +02:00
Ricardo Pardini
61d85648ea cleaning: stop lying about CLEAN_LEVEL=debs -- now it simply deletes every .deb in the folder (same as =alldebs) 2023-03-31 09:12:44 +02:00
Ricardo Pardini
ed0b406265 lib: fix: replace undue export statements with declare -g 2023-03-31 09:12:44 +02:00
Ricardo Pardini
fb4477afb8 oras: add --insecure to further retries if we get a failure on first try, to workaround GitHub's March/2023 mess with certificates 2023-03-31 09:12:44 +02:00
Ricardo Pardini
a77d5f3c70 fix: declare some variables that were leaking into global namespace for no reason
- especially `for VAR in ...` and VAR leaks sans the declare
- rename `fragment_manager_cleanup_file` to `extension_manager_cleanup_file`
2023-03-31 09:12:44 +02:00
Ricardo Pardini
02c561ce0e bat-cat: introduce tooling support for bat (colorized/smart cat), including DEBIAN/xxx syntaxes; cached in Docker image 2023-03-31 09:12:44 +02:00
Ricardo Pardini
b820a4d68f tools: oci-oras/shellcheck: fix, test if binary already in place before considering copying it, but only after BIN is actually defined 2023-03-31 09:12:44 +02:00
Ricardo Pardini
2f4f117000 prepare ORAS tooling early, so we don't get spurious failures "only on first build" 2023-03-17 08:32:11 +01:00
Ricardo Pardini
630ee512cb stop the build if LIB_TAG is set at all; some users expect it to work while it doesn't 2023-03-17 08:32:11 +01:00
Ricardo Pardini
6b965c605c completely remove any traces of KERNEL_ONLY and exit with error if set 2023-03-17 08:32:11 +01:00
Igor Pečovnik
b7b8eb7b72
Add / modify (c) in bash scripts (#4922)
* Add / modify (c) in bash scripts

Signed-off-by: Igor <igor@armbian.com>

* Add (c) to the source config files

---------

Signed-off-by: Igor <igor@armbian.com>
2023-03-09 18:30:40 +01:00
Vyacheslav Bocharov
123b31b161 Replace absolute path to git-tree with relative, so it is accesible without docker builder. 2023-02-28 09:26:12 -03:00
Ricardo Pardini
c2004faf9c
git-ref2info: fix https Makefile URL for gitlab.com (eg, rk35xx-legacy) (#4848) 2023-02-22 20:59:56 +01:00
Ricardo Pardini
26783b050f
armbian-next: pip3: workaround new problems introduced by pip 23+ 2023-02-18 07:46:47 -03:00
Ricardo Pardini
85f3fb5c93
armbian-next: oci-oras: don't capture our own debugs as part of the manifest (d'oh -- thanks RichN) 2023-02-18 07:46:45 -03:00
Ricardo Pardini
2350c5697f
armbian-next: cleaning: deprecate CLEAN_LEVEL=debs -- doesn't do anything, just shows what would be cleaned 2023-02-18 07:46:16 -03:00
Ricardo Pardini
e46233084a
armbian-next: memoize: expire all memoized files after 1 hour 2023-02-18 07:46:15 -03:00
Ricardo Pardini
3d8ee5ce38
armbian-next: oras: add 3/5 retries to oras pull/push 2023-02-18 07:46:11 -03:00
Ricardo Pardini
8694daad78
armbian-next: chroot-helpers: mount_chroot()/umount_chroot(): add warning with stacktrace if called sans logging section 2023-02-18 07:46:05 -03:00
Ricardo Pardini
26f0700da8
armbian-next: chroot-helpers: if unable to mount devpts, use a mount bind from host
Co-authored-by: Paolo Sabatino <paolo.sabatino@gmail.com>
2023-02-18 07:46:04 -03:00