Commit Graph

121 Commits

Author SHA1 Message Date
Igor Velkov
ae6735f470
patching: fix #9028 timestamp when multiple patches touch same file (#9489)
If patch B sorts after patch A but has an older mtime, it would
overwrite A's timestamp on the shared file, causing the kernel Makefile
to skip recompilation.

Fix: only call os.utime() when the new mtime is strictly greater than
the file's current mtime.
2026-03-05 10:13:40 +01:00
Ricardo Pardini
b48c253f67 inventory-boards-csv: report ATFSOURCE and ATFBRANCH 2026-01-15 20:17:43 +01:00
Ricardo Pardini
1ab9073db2 patching: Makefile Autopatcher: only add entries for DTs in same target_dir
- when auto-patching more than one Makefile, only add the DTs added to the corresponding directory
2026-01-09 20:29:08 +01:00
Ricardo Pardini
a85a5aef67 patching: introduce add-only mode for DT Makefile AutoPatcher
- Autopatcher was commiting atrocities to the upstream DTs that are so
  carefully hand-crafted, by simply rewriting everything, or trying to be
  smart and failing at it (`incremental: true`)
- Instead, add `add-only: true` mode, where we simply only ever add to the
  end of the Makefile, adding each new .dts and optionally the `subdir-y`
  for overlays
2026-01-09 20:29:08 +01:00
Igor Pecovnik
4d60ce08f2 chore: update copyright years to 2026
Update all copyright notices in shell scripts from 2025 to 2026.

## Changes

- **Igor Pecovnik**: 2013-2025 → 2013-2026 (129 files)
- **Ricardo Pardini**: 2023-2025 → 2023-2026, 2020-2025 → 2020-2026 (5 files)

## Additional Improvements

Also updated the backtitle in `lib/functions/configuration/interactive.sh`:
- Changed title from "Armbian building script" to "Armbian Linux build framework"
- Removed docs link for cleaner display
- Uses dynamic year calculation with separate declaration (fixes shellcheck SC2155)
2025-12-25 12:03:34 +01:00
tabris
76403b4d3f build machinery - replace wget with curl cuz reasons 2025-10-17 19:33:20 +02:00
igorpecovnik
a1c49e4e5f tools: Bump SHELLCHECK_VERSION from 0.10.0 to 0.11.0 2025-08-16 21:15:24 +08:00
igorpecovnik
5efa4a16df tools: Bump SHELLFMT_VERSION from 3.11.0 to 3.12.0 2025-07-18 17:43:52 +02:00
Ricardo Pardini
484aac7df2 lib/tools/info: mapper-oci-uptodate: adapt to oras-py 0.2 api change 2025-04-12 20:06:13 +02:00
Ricardo Pardini
9e5ca480d2 lib/tools: Python CLI for ORAS (add non-existing case) 2025-04-12 20:06:13 +02:00
Ricardo Pardini
5d712c242a lib/tools: Python CLI for ORAS (bump to 0.2.y, api change, not too bad) 2025-04-12 20:06:13 +02:00
Ricardo Pardini
89eae213a7 lib/tools: Python CLI for ORAS (0.1.y) 2025-04-12 20:06:13 +02:00
igorpecovnik
19b0a062a7 tools: Bump SHELLFMT_VERSION from 3.10.0 to 3.11.0 2025-03-16 11:53:12 +01:00
igorpecovnik
68c758f4a1 tools: Bump SHELLFMT_VERSION from 3.9.0 to 3.10.0 2024-11-03 22:24:23 +01:00
Ricardo Pardini
5396ffb602 patching: emit warnings when files in dt folder overwrite pre-existing files (DTs that landed upstream)
- it's more and more common that the (bare) DT files in our `dt` folders have landed upstream
- this adds warnings and marks the patching table red when some bare-dt file overwrites what's already in git
- without this it's very easy to forget them there during bumps
2024-09-18 16:31:34 +02:00
igorpecovnik
46dd7d2760 tools: Bump SHELLFMT_VERSION from 3.8.0 to 3.9.0 2024-09-16 13:17:49 +02:00
amazingfate
24be131989 dt_makefile_patcher: fix dtbs with dot in filename 2024-09-10 10:48:53 +08:00
ColorfulRhino
39fdf21b98
Beautify shellfmt and add board configs to formatting list (#6910)
* tools: shellfmt: Beautify and add board configs to formatting list
* Format the whole code using `lib/tools/shellfmt.sh`
---------
Co-authored-by: Igor <igor@armbian.com>
2024-07-08 17:00:38 +02:00
ColorfulRhino
308eba5d31 cli: dt_makefile_patcher: Allow numbers in configopt regex
This fixes detection of the config option for SpacemiT K1 which is `CONFIG_SOC_SPACEMIT_K1X`.
Otherwise it would not find it because it includes a '1'.
2024-07-01 19:15:00 +02:00
ColorfulRhino
7f8576f378 tools: Slightly improve output for shellfmt 2024-06-25 07:06:34 +02:00
ColorfulRhino
a46481c96a tools: Update shellfmt default version to 3.8.0 2024-06-25 07:06:34 +02:00
Ricardo Pardini
282fb964e5 patching: rewrite: fully stabilize patch index stanzas as well as From lines
- `git format-patch --zero-commit` doesn't affect `index xxx...yyy` lines, only `From: `
  - so use the _classy_ "use a regex with a callback" solution as git format-patch doesn't offer one
- this will make _all_ patches change when rewritten, but hopefully _for the last time_ !
- we need to preserve `index 000000000000..xxx` as zeros, which indicate new file creation, thus:
  - new file creations are rewritten as `index 000000000000..111111111111`
  - non-creations are rewritten as `index 111111111111..222222222222`
- this is the final version of #6455
2024-06-09 10:42:26 +02:00
ColorfulRhino
42874201c4 cli: Update shellcheck 0.9.0 --> 0.10.0 2024-03-28 14:07:32 +01:00
ColorfulRhino
7d38b4273a cli: rewrite-<uboot,kernel>-patches-needing-rebase 2024-03-13 21:01:26 +00:00
Ricardo Pardini
bf7f20776c cli: introduce inventory-boards cli command, for hopefully painless & useful one-board-per-line CSV output
- `inventory-boards-csv.py`: moar Python atrocities done to data; idea here is to output one line per board, with <<varies>> for fields that, well, vary (mainly across BRANCH'es)
- `inventory-boards` sets `TARGETS_FILE` to something that doesn't exist, so the `default-targets.yaml` is used (so same list for everyone, save for userpatched-boards)
2024-03-06 00:58:26 +01:00
Ricardo Pardini
911e71241b patching: fix: use correct has_overlay_directories instead of has_dts_directories for overlay processing 2024-03-02 20:48:32 +01:00
Darsey Litzenberger
ce33bddb58 Make the colorized build output readable on a light background
This fixes the readability of colorized output of compile.sh when
running on a terminal with a light background.  It uses the COLORFGBG
environment variable similarly to how the ip(8) command does.

Signed-off-by: Darsey Litzenberger <dlitz@dlitz.net>
2024-02-09 19:38:03 +01:00
Ricardo Pardini
86cfcf304a patching: support 'incremental' Makefile auto-patching, for vendor kernels
- without incremental: true:
  - we replace the DT Makefile based on *.dts at the DT directory after all patching/copying is done
  - overlay target is added if overlays present
- with incremental: true:
  - we parse the Makefile, parse the DTs being built, and only add .dts's we have added from 'dt' directory
  - overlays target are not added; we assume smth else already added them
2024-02-04 11:24:31 +01:00
amazingfate
afc30e64eb patching: set commit index length to 12 when rewriting patches 2023-12-30 13:33:56 +01:00
Ricardo Pardini
6186a41c2b patching: if 0 files left to commit after config-filtering, don't commit nor rewrite the patches
- this is mostly useful for Makefile's and DT patches which should end be bare instead of null-patched
2023-10-26 14:48:49 +00:00
Ricardo Pardini
35b4859b9c pipeline: parallel (threaded) mapper-oci-uptodate.py 2023-10-14 21:49:52 +02:00
Ricardo Pardini
568ed3ae39 pipeline: don't bomb when logging int value's length (GHA outputs) 2023-10-13 21:05:33 +02:00
Ricardo Pardini
ed39cb04fb pipeline: add hard 17*30 limit to number of total matrix jobs
- show size of each GHA output in logs
2023-10-13 20:13:10 +02:00
Ricardo Pardini
3b54d17480 patching: accept a sha1 in BASE_GIT_TAG (as well as branch or tag)
- just for convenience
2023-10-13 18:04:14 +02:00
Ricardo Pardini
1c5add5e51 don't break downloads if GITHUB_SOURCE is unset
- fixes f200587b86 from https://github.com/armbian/build/pull/5791
2023-10-10 20:25:53 +02:00
iysheng
f200587b86 Use GITHUB_SOURCE to accelerate download files from github 2023-10-08 16:45:00 +02:00
Ricardo Pardini
ef01d98347 patching: show auto-patch'ed in summary table (in blue); remove spurious warnings 2023-10-03 14:10:46 +02:00
Ricardo Pardini
a547d3b101 patching: show the patch number for multi-patch mbox files in summary table 2023-10-03 14:10:46 +02:00
Ricardo Pardini
50895a24ee patching: enhance the DT Makefile Autopatcher to work with armhf's new 6.5 subdir Makefiles as well as arm64
- arm64: one-rule-per-dtb
- arm32: single-rule, multi-dtb
2023-10-03 14:10:46 +02:00
Ricardo Pardini
9324d1ec82
pipeline: output-gha-matrix: add self_hosted (yes/no) and fetch_depth (0/1) (#5750)
```
# For each item in chunk, check if it is going to run in a GH-hosted runner or self-hosted, and set some matrix variables
# accordingly; self_hosted: yes/no, and more specifically, fetch_depth, which is 0 for self-hosted, and 1 for GH-hosted.
# The reasoning for this is that git clones are much faster if not shallow on self-hosted, but much slower on GH-hosted.
```

- `self_hosted` -> `shosted` for brevity in JSON matrix
- `fetch_depth` -> `fdepth` for brevity in JSON matrix
2023-09-29 16:55:09 +02:00
Ricardo Pardini
800d92045d tools: patching: don't rewrite auto-generated patches (eg, drivers) 2023-09-29 08:48:01 +02:00
Ricardo Pardini
d402092682 pipeline: use ncores * 4 for info gathering 2023-08-23 21:35:02 +02:00
Ricardo Pardini
274d394ab2 pipeline: handle exit_with_target_not_supported_error() (retcode 44) as warning and not error
- split errors and warnings into multiple lines, so a bit easier to see in the logs what the real error was
- also turn down a few spurious warnings to debugs
2023-08-23 21:35:02 +02:00
Ricardo Pardini
688e415832 pipeline: add userspace inventory capabilities
- digs into config/distributions and config/desktops for info
- this produces `output/info/all_userspace_inventory.json`
  - this is now passed down to the `targets-compositor` in `cli-jsoninfo`
- `targets-compositor` now accepts `userspace:` as `items-from-inventory`
- extra: add `targets-composed` CLI command, to stop after targets-compositor
2023-08-23 21:35:02 +02:00
Ricardo Pardini
bd5a577e40 pipeline: streamline output-debs-to-repo-json to pass down info and avoid parsing paths completely 2023-08-12 09:58:32 +02:00
Ricardo Pardini
4624b581da pipeline: force compositor to include REVISION= just like it already did for BETA=
- this way the prepare step's REVISION (possibly passed via CLI parameter) is always sovereign
2023-08-12 09:58:32 +02:00
Ricardo Pardini
a876d0f882 pipeline: output-debs-to-repo-json.py: adapt to new reversioned JSON info 2023-08-12 09:58:32 +02:00
Ricardo Pardini
0251d65b7a repo-reprepro.py: set -e & pipefail 2023-08-12 09:58:32 +02:00
Ricardo Pardini
ee3059eb2f pipeline: debs-to-repo-download is now done in parallel; default cores*2 limited to 16 or PARALLEL_DOWNLOADS_WORKERS
- also avoid log archiving during individual download jobs via SKIP_LOG_ARCHIVE=yes
- I've tested with PARALLEL_DOWNLOADS_WORKERS=16 and it saturates my gigabit link, ghcr.io is great at reads
- more than ~16-ish might be too much though
2023-07-27 21:32:19 +02:00
Ricardo Pardini
b92575381a pipeline: inventory all board vars; add not-eos-with-video; introduce TARGETS_FILTER_INCLUDE
> How to use:
>
> `./compile.sh inventory` - does just the board inventory; look for output in `output/info`
>
> `./compile.sh targets-dashboard` - does inventory, targets compositing, and images info; look for output in `output/info`, read the instructions output by the command if you want to load the OpenSearch dashboards.
>
> `./compile.sh targets` - does the full targets compositing and artifacts, look for output in `output/info`
>
> If you don't have a `userpatches/targets.yaml`, _one will be provided for you_ defaulting to Jammy minimal CLI
> and Jammy xfce desktop, for all boards in all branches. You can pass filters via `TARGETS_FILTER_INCLUDE=...` to narrow.
>

- board JSON inventory:
  - more generic regex parsing of variables from board files:
    - all top-level (non-indented) variables are parsed and included in the JSON board inventory
    - this allows us to add new variables to the board files without having to update the parser
    - variables can be bare, `export` or `declare -g`, but **_must_ be quoted** (single or double) and UPPER_CASE
  - some special treatment for certain variables:
    - `KERNEL_TARGET` is parsed as a _comma-separated_ list of valid BRANCH'es
    - `BOARD_MAINTAINER` is parsed as _space-separated_ list of valid maintainer GH usernames as `BOARD_MAINTAINERS: [...]` in the JSON
      - script complains if `BOARD_MAINTAINER` is not set in core boards. Empty is still allowed.
    - `HAS_VIDEO_OUTPUT="no"` causes `BOARD_HAS_VIDEO: false` in the JSON (for desktop-only inventorying, see below)
- introduce `not-eos-with-video` in `items-from-inventory` at the targets compositor
  - the same as `not-eos`, but with added `BOARD_HAS_VIDEO: true` filter, see above
- introduce `TARGETS_FILTER_INCLUDE` for targets compositor
  - this filters the targets _after_ compositing (but before getting image info), based on the board inventory data
  - it's a comma-separated list of `key:value` pairs, which are OR-ed together
  - new virtual info `BOARD_SLASH_BRANCH` post-compositing inventory for filtering of a specific BOARD/BRANCH combo (e.g. `odroidhc4/edge`)
  - some interesting possible filters:
    - `TARGETS_FILTER_INCLUDE="BOARD:odroidhc4"`: _only_ build a single board, all branches. JIRA [AR-1806]
    - `TARGETS_FILTER_INCLUDE="BOARD_SLASH_BRANCH:odroidhc4/current"`: _only_ build a single board/branch combo
    - `TARGETS_FILTER_INCLUDE="BOARD:odroidhc4,BOARD:odroidn2"`: _only_ build _two_ boards, all branches.
    - `TARGETS_FILTER_INCLUDE="BOARD_MAINTAINERS:rpardini"`: build all boards and branches where rpardini is a maintainer
    - `TARGETS_FILTER_INCLUDE="BOARDFAMILY:rockchip64"`: build all boards and branches in the rockchip64 family
  - image-info-only variables like `LINUXFAMILY` is **not** available for filtering at this stage
- rename `config/templates` `targets-all-cli.yaml` to `targets-default.yaml`
    - this is used when no `userpatches/targets.yaml` is found
    - new default includes all boards vs branches for non-EOS boards
        - also desktop for all boards that _don't_ have `HAS_VIDEO_OUTPUT='no``
- introduce simplified `targets-dashboard` CLI:
  - does only inventory, compositing, and image info, but not artifact reducing, etc.
  - ignore desktop builds in the OpenSearch indexer
  - update the OpenSearch Dashboards, including new information now available
- invert the logic used for `CLEAN_INFO` and `CLEAN_MATRIX`
    - defaults to `yes` now, so new users/CI don't get hit by stale caches by default
    - repo pipeline CLI stuff is usually run on saved/restored artifacts for `output/info`, so don't clean by default via the CLI
2023-07-26 15:15:02 +02:00