Commit Graph

141 Commits

Author SHA1 Message Date
Ricardo Pardini
fd7003125e shellcheck: skip running shellcheck, with a warning, if running on riscv64
- someone go add riscv64 to https://github.com/koalaman/shellcheck
2024-04-07 13:23:34 +02:00
Ricardo Pardini
9bef0a5e5a oci-oras: use ORAS 1.2.0-beta.1 for riscv64 2024-04-07 13:23:34 +02:00
Ricardo Pardini
71753f4b3b python-tools: bump PyYAML to 6.0.1 to fix build failures on sid/trixie/noble etc 2024-03-03 16:07:19 +01:00
Darsey Litzenberger
ce33bddb58 Make the colorized build output readable on a light background
This fixes the readability of colorized output of compile.sh when
running on a terminal with a light background.  It uses the COLORFGBG
environment variable similarly to how the ip(8) command does.

Signed-off-by: Darsey Litzenberger <dlitz@dlitz.net>
2024-02-09 19:38:03 +01:00
Gunjan Gupta
218f54dc28 Fix git_ensure_safe_directory function again
The command added to check whether the directory is git or not was
having syntax error. Fixed the same. Also it seems adding safe-directory
with local doesn't really work which actually makes sense. hence
reverting to using global instead. Its still safer than before as
we are not adding * and are adding the actual path of the directory
2024-02-03 04:50:42 +05:30
Igor Pecovnik
6b42f34b73 Add additonal check before running git command 2024-01-26 22:11:14 +01:00
Igor Pecovnik
12910e0d37 Fixes poorly made Git safe.directory workaround 2024-01-26 21:28:05 +01:00
amazingfate
64094ee20e fix gitee mirror 2024-01-22 21:17:15 +01:00
Igor Pecovnik
18ba950f38 Enable support for extensions within USERPATCHES_PATH 2023-12-29 23:37:10 +01:00
Andrew Davis
33d8026508
Use TI kernel for K3 family with current v6.1.y (#5999)
* Add git.ti.com to memoized_git_ref_to_info() as a generic cgit

Like kernel.org, git.ti.com uses cgit. We might want to change this logic to
accept the git hosting type (cgit, gitlab, github, etc..) and not match only
on known repo urls. For now just add this repo url to the known list.

* K3: Switch current to TI vendor kernel v6.1.y

* K3: Bump edge to kernel v6.6.y
2023-12-05 12:50:40 +01:00
Ricardo Pardini
67b3e0dd07 git: fetch_from_repo(): introduce FETCH_FROM_REPO_CALLBACK_IF_REF_MUTABLE
- function defined in FETCH_FROM_REPO_CALLBACK_IF_REF_MUTABLE is called if reference is mutable
  - which is anything that's not `commit:` or `tag:`
    - and really there's no guarantee that `tag:` is immutable, but lets assume
2023-11-25 13:38:10 +01:00
amazingfate
81c460e3bf use GHPROXY_ADDRESS to customize ghproxy url 2023-11-21 17:06:14 +01:00
Ricardo Pardini
1e184f14af core extensions: fix: don't redefine extension_function_info when recursing
- otherwise when there's a chain of enable_extension() we lose the original info
2023-11-19 22:31:33 +01:00
Ricardo Pardini
91094149de base-files / apt-utils: stop trying non "-updates" repos when looking for Ubuntu LTS release package
- otherwise, when the lookup at "jammy-updates" fails (due to server instability), and "jammy" works, we end up with the wrong version
- non-LTS releases don't have the "-updates" repo, so don't even try, which should make everything faster
- TODO: yet-another opportunity to have a release metadata file, otherwise this is yet-another place where we list "jammy"
2023-10-21 13:58:37 +02:00
Ricardo Pardini
1c5add5e51 don't break downloads if GITHUB_SOURCE is unset
- fixes f200587b86 from https://github.com/armbian/build/pull/5791
2023-10-10 20:25:53 +02:00
iysheng
f200587b86 Use GITHUB_SOURCE to accelerate download files from github 2023-10-08 16:45:00 +02:00
Ricardo Pardini
000f491617 kernel/memoize: allow customizing memoize's default TTL of 3600 seconds; allow customizing kernel TTL via KERNEL_GIT_CACHE_TTL 2023-09-29 14:25:58 +02:00
Andrew Davis
75bbc299fc extensions: Fix dump_extension_method_sources_functions() to iterate
The function dump_extension_method_sources_functions() dumps the content
of a single argument function. It is used in several spots were an array
is passed instead. We could fix all these call sites to iterate the call,
or we could change the called function to do the iteration. Do the latter.
2023-09-24 20:18:47 +02:00
amazingfate
1eaf25ad3d add mainline kernel mirror support 2023-08-23 21:34:48 +02:00
Ricardo Pardini
11df81403d config/lib/exts: run shellfmt
- `bash lib/tools/shellfmt.sh`
- mostly spacing changes that accumulated over time
2023-08-22 14:28:14 +02:00
Igor
2a8cb5793f Removing exceptions needed to support riscv64 withing Debian
Signed-off-by: Igor <igor@armbian.com>
2023-08-17 22:09:05 +02:00
Gunjan Gupta
bc0e131747 Enforce use of bind mount for /dev within chroot
The /dev/mapper directory created by devtmpfs lacked entry for
armbian-root thereby breaking the cryptsetup configuration generated
within initrd file. Use bind mount as that doesn't seem to suffer from
that issue.
2023-07-22 02:18:49 +05:30
Lane Jennison
ad97c1b7ee rockchip-rk3588 collabora: new branch (by @lanefu)
- `bsdextrautils` for kernel 6.4 (only if collabora branch)
- kernel 6.4 major, but skip rtw88 driver patching
2023-07-01 14:50:35 +02:00
Jens J
42de8b678c handle kernel.org forks 2023-06-26 13:45:04 +02:00
Andrew Davis
92aa7b3de2 Use HTTPS for kernel.org sources checkout
HTTPS is more secure and less likely to be blocked by firewalls. The
other sources here use HTTPS now, update the kernel to the same.
2023-06-14 15:55:09 +02:00
Ricardo Pardini
70a64a0415 patching: Patching Summary and Patching Failures tables using Python's Rich
- even Rich'er patch output by colorizing certain strings green/yellow/red
- BASE_GIT_TAG now very sneakily also accepts a branch name
- IMPORTANT: this includes: BREAKING CHANGE: patches failing to apply now break the build. fixes #4958
  - also break on legacy `process_patch_file()` failure, remove `EXIT_PATCHING_ERROR`
2023-06-13 12:29:52 +02:00
Ricardo Pardini
31131fe14e bsp-cli/BETA: fixes for bsp-cli bootscript hashing; don't normalize bootscript vars; debugging for BETA=yes/no stuff
also:
- get rid of comments and `SUBREVISION` and `RC` variables
- better debugging in calculate_hash_for_variables() do_normalize_src_path=yes/no
- tag a few places where output/debs might wrong in face of BETA=yes/no
2023-05-25 19:30:40 +02:00
Ricardo Pardini
15fcc0bb00 artifacts: introduce calculate_hash_for_variables() which normalizes the input; use in kernel/uboot/etc
- some hashed variables might contain "${SRC}", so hashes never match, unless built in Docker
  - this strips away SRC from all vars and adds debugging so we can detect more later.
- `artifact-uboot`: include more variables into hash, for ATF & rk stuff
- `artifact-armbian-desktop`:
  - hashed vars actually contain /armbian in different context, skip normalization in this case
  - include results of relevant aggregation into artifact_input_variables
    - otherwise: desktops with different appgroups/configs lead to build failures in pipeline
    - will cause warnings in JSON preparation step, if more than one appgroups/config combo is targeted, since repo can only have one
2023-05-24 19:23:08 +02:00
Ricardo Pardini
7d72d18bb0 git-ref2info: source.denx.de is a GitLab instance, add it to the selector; fixes #5168 2023-05-23 16:34:40 +02:00
Igor
e67215bd6a URL for riscv64 is different then expected
https://deb.debian.org/debian-ports/pool -> pool-riscv64
2023-05-21 11:11:07 +02:00
Ricardo Pardini
1fd5d519d4 introduce armbian-base-files artifact, which downloads & repacks base-files from upstream distro
- bsp-cli: now depends on `base-files (>= ${REVISION})`, this way upgrading the bsp-cli causes our base-files to be installed
  - bsp-cli no longer does gymnastics with /etc/os-release et al, all done in armbian-base-files now
- general/apt-utils.sh: introduce `apt_find_upstream_package_version_and_download_url()`
- base-files: add release to version, in order to comply with repo restrictions (valid repos can't have two different debs with same name and version, md5 must match)
2023-05-18 20:49:06 +02:00
Ricardo Pardini
e3a4060595 github-actions: more logging for GHA actions outputs 2023-05-16 20:09:47 +02:00
Ricardo Pardini
27a021d193 oci-oras: retry harder, and sleep for more time, if push failed 2023-05-16 20:09:47 +02:00
Ricardo Pardini
b456554610 artifacts(debs): rationalize "bash hash" creation with new calculate_hash_for_bash_deb_artifact() helper 2023-05-16 20:09:47 +02:00
Ricardo Pardini
78700cf139 logging/retries: curb CI/GHA warning emitted _for each retry_ 2023-05-15 17:43:25 +02:00
Ricardo Pardini
23a2b34847 pipeline: add output-gha-workflow-template.py utility, for rendering GHA workflow YAML templates with chunks
- python-tools: add Jinja2. Incredible how we made it this far without it.
- output-gha-workflow-template: a double-templater, first runs jinja with a custom syntax, then "moar magic" to be useful for GHA
2023-05-03 07:49:53 +02:00
Ricardo Pardini
f992dcd466 git-ref2info: retries for getting Makefile body (https://raw.githubusercontent.com has been failing a lot) 2023-05-01 22:47:03 +02:00
Ricardo Pardini
f8ddf7f9e2 🔥 JSON info pipeline: v18
- pipeline: add `pipeline` context object to targets; use it to filter artifacts and images to build; warn about oci-name with multiple oci-tags
- pipeline: better error messages when info's fail; show some (out-of-order) progress messages during parallel info gathering
- pipeline: targets-compositor: add `not-eos` inventory
- TARGETS_FILENAME, log all OCI lookups
- SKIP_IMAGES
- IMAGES_ONLY_OUTDATED_ARTIFACTS
- no dash in chunk id in JSON
- pipeline: very initial chunking, using the same outputs
- pipeline: template targets, `items-from-inventory:` inventory expansion, CHECK_OCI=yes, CLEAN_MATRIX=yes, CLEAN_INFO=yes, many fixes
- cli: `inventory` / `targets` / `matrix` / `workflow`
- pipeline: workflow beginnings
- pipeline: general log cleanup + OCI stats / better miss handling
- pipeline: fixes/reorg
- pipeline: catch & log JSON parsing errors
- pipeline: gha matrix: use IMAGE_FILE_ID as job description
- pipeline (delusion): gha workflow output, based on old matrix code
- pipeline: better parsing and reporting of stderr log lines (under `ANSI_COLOR=none`)
- pipeline: mapper-oci-uptodate: use separate positive/negative cache dirs (GHA will only cache positives); cache negs for 5 minutes locally
- pipeline: output-gha-matrix artifacts + images
  - pipeline: output-gha-matrix artifacts + images: "really" and fake 1-item matrix if empty
- pipeline: move files into subdir; update copyright & cleanup
- pipeline: refactor bash jsoninfo driver a bit
- pipeline: outdated-artifact-image-reducer
- pipeline: introduce `target_id` at the compositor, aggregate it at the reducer, carry it over in the artifact info mapper
- pipeline: mapper-oci-uptodate
- pipeline: info-gatherer-artifact, with PRE_PREPARED_HOST
- pipeline: refactor/rename info-gatherer-image.py
- pipeline: beginnings
2023-05-01 22:46:25 +02:00
Ricardo Pardini
59d6690a11 git-ref2info: many fixes for nonstandard repos; try a few different dereference and refs/ combinations in a certain order; remove codeaurora repo handling
- git-ref2info: when resolving SHA1's for 'xxx' tag or branch, try first `refs/heads/xxx`, then just 'xxx'; validate SHA1 with 40-char regexp
  - this works-around Gerrit-style repositories that have a 'refs/for/xxx' ref that gets in the way
  - case in point: https://github.com/hardkernel/u-boot/tree/odroidc-v2011.03 (has both refs/heads/odroidc-v2011.03 and refs/for/odroidc-v2011.03)
- git-ref2info: when processing tags for a SHA1, try both dereference and non-dereferenced, in that order, to get a SHA1
  - this solves old problem reported in #4916: GitHub incorrectly processes dereference requests for tags that are not actually annotated
  - case in point: https://github.com/ayufan-rock64/linux-kernel/releases/tag/4.4.202-1237-rockchip-ayufan
- `codeaurora` is no more, https://bye.codeaurora.org/
2023-04-28 21:32:52 +02:00
amazingfate
97a2e41f62 use common sed for GITHUB_SOURCE replace 2023-04-21 14:21:28 +02:00
Ricardo Pardini
8cd830d6e1 memoize-cached: add flock locking to run_memoized() to avoid error when run in parallel 2023-04-18 09:27:38 +02:00
amazingfate
a0f59bf914 use ghproxy to accelerate memoized_git_ref_to_info 2023-04-17 13:42:53 +02:00
Ricardo Pardini
9b229f25ad artifact-uboot: add hashes for extension and old-timey -Hooks and -Variables hash (BOOTDELAY et al)
- helper for hashing old-timey hooks: `calculate_hash_for_function_bodies()` in `hash-files.sh` cos I'm lazy
2023-04-03 21:21:03 +02:00
Ricardo Pardini
d2bb1f893a extensions: many fixes; allow for dumping the source of hooks, via dump_extension_method_sources_functions() and dump_extension_method_sources_body()
- fix: indent of generated sources
- fix: no reason to source the generated file in a loop. do it once. 20x faster?
- `dump_extension_method_sources_functions "name_of_hook"`: get function definitions (inside braces)
- `dump_extension_method_sources_body "name_of_hook"`: get the function bodies (sans-braces)
2023-03-31 09:12:44 +02:00
Ricardo Pardini
61d85648ea cleaning: stop lying about CLEAN_LEVEL=debs -- now it simply deletes every .deb in the folder (same as =alldebs) 2023-03-31 09:12:44 +02:00
Ricardo Pardini
ed0b406265 lib: fix: replace undue export statements with declare -g 2023-03-31 09:12:44 +02:00
Ricardo Pardini
fb4477afb8 oras: add --insecure to further retries if we get a failure on first try, to workaround GitHub's March/2023 mess with certificates 2023-03-31 09:12:44 +02:00
Ricardo Pardini
a77d5f3c70 fix: declare some variables that were leaking into global namespace for no reason
- especially `for VAR in ...` and VAR leaks sans the declare
- rename `fragment_manager_cleanup_file` to `extension_manager_cleanup_file`
2023-03-31 09:12:44 +02:00
Ricardo Pardini
02c561ce0e bat-cat: introduce tooling support for bat (colorized/smart cat), including DEBIAN/xxx syntaxes; cached in Docker image 2023-03-31 09:12:44 +02:00
Ricardo Pardini
b820a4d68f tools: oci-oras/shellcheck: fix, test if binary already in place before considering copying it, but only after BIN is actually defined 2023-03-31 09:12:44 +02:00