Commit Graph

27 Commits

Author SHA1 Message Date
Igor Velkov
1b74748622
Extension: ccache-remote — shared compilation cache via Redis/HTTP (#9369)
* add hook to allow customizing before kernel make env creation
* Hook runs in docker_cli_prepare_launch() just before DOCKER_EXTRA_ARGS
is processed, allowing extensions to add Docker arguments with a more
descriptive hook name than add_host_dependencies.
* Extension: ccache-remote

Enables ccache with remote Redis storage for sharing compilation cache across build hosts.

Features:
- Auto-discovery via Avahi/mDNS (ccache.local hostname)
- Explicit Redis server configuration via CCACHE_REMOTE_STORAGE
- Build statistics display at end of build (hit/miss/error rates)
- Support for both Docker and native builds
- Hooks for kernel and u-boot compilation environments

Documentation includes server setup instructions with security warnings,
client mDNS configuration, and cache sharing requirements.


* uboot: fix ccache environment and add extension hook

U-Boot build uses `env -i` which clears all environment variables.
CCACHE_DIR and CCACHE_TEMPDIR were not explicitly passed to make,
unlike kernel build (kernel-make.sh). This caused ccache to use
default directory instead of configured Armbian one, breaking
cache statistics and shared cache functionality.

Changes:
- Add CCACHE_DIR and CCACHE_TEMPDIR to uboot_make_envs
- Add uboot_make_config hook for extensions (similar to kernel_make_config),
  allowing modification of environment variables before compilation

* add long list of allowed ccache-related env vars
* set permissions to ccache files RW for everyone if cache not private
* ccache: add ccache_post_compilation hook for extensions
* ccache-remote: use ccache_post_compilation hook instead of cleanup handler

Show remote ccache stats after each compilation (kernel, uboot) via hook,
instead of once at the end via cleanup handler. Stats now shown even on
build failure.

* ccache: show stats with safe arithmetic
* ccache/uboot: improve code comments per review feedback

- uboot.sh: clarify ARMBIAN=foe workaround for dual-compiler scenario
- ccache-remote.sh: document that CCACHE_REDIS_CONNECT_TIMEOUT must be
  set before extension loads

* ccache-remote: mask storage URLs in logs

Mask CCACHE_REMOTE_STORAGE when emitting Docker env debug logs.

* ccache-remote: extract ccache_inject_envs() helper to deduplicate passthrough loops

Extract ccache_inject_envs() helper to deduplicate identical passthrough
loops in kernel and uboot make config hooks.

ccache-remote: rename functions to follow project naming conventions

Rename get_redis_stats and mask_storage_url to ccache_get_redis_stats
and ccache_mask_storage_url to follow project naming conventions.

ccache-remote: mask credentials in debug log output for passthrough loops

Mask CCACHE_REMOTE_STORAGE value through ccache_mask_storage_url() before
logging in both Docker env and make env passthrough loops to avoid leaking
credentials into build logs.

* ccache-remote: add HTTP/WebDAV backend and DNS discovery
* ccache-remote: move extension script into directory layout
* ccache-remote: add server setup docs and config files
* ccache-remote: validate Redis credentials in URLs
* ccache-remote: document Redis auth options and safe passwords

Add separate insecure config example for trusted networks.

Recommend URL-safe hex passwords and update setup docs.

* ccache-remote: improve Docker loopback handling and IPv6 host parsing
2026-03-01 01:18:35 +01:00
Igor Velkov
b7bc3a770d clang: enable colored diagnostic output for kernel compilation
When building kernels with KERNEL_COMPILER=clang, compiler warnings
were displayed without color despite -fdiagnostics-color=always being
set in KCFLAGS. This GCC-native flag is not reliably honored by clang
when invoked through ccache and the kernel build system with LLVM=1.

Add -fcolor-diagnostics (clang's native flag) to the clang-specific
extra_warnings to ensure colored warning output.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-31 17:12:25 +01:00
Igor Velkov
a661744d1d kernel: add custom_kernel_make_params extension hook
Add a new extension hook point in run_kernel_make_internal() that allows
extensions to modify kernel make parameters before compilation.

Extensions can now modify:
- common_make_params_quoted - parameters passed to make
- common_make_envs - environment variables for make

This enables features like CROSS_COMPILE_COMPAT for 32-bit compat vDSO
on arm64 builds without modifying core build scripts.

Refs: https://github.com/armbian/build/issues/9216

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-25 11:36:20 +01:00
Ricardo Pardini
acfd9c8dcb "get completely rid of dead code toolchain stuff", pt2
- this stuff has been laying around, unused, for years
2026-01-14 20:08:21 +01:00
Ricardo Pardini
6889c8c0a4 drop find_toolchains, SKIP_EXTERNAL_TOOLCHAINS and all $toolchain PATH injections
- we've had SKIP_EXTERNAL_TOOLCHAINS=yes for ~5 years now
- drop all usages, mostly through `find_toolchains()`
- drop all manual PATH env injections (we've centralized if ever needed)
2026-01-14 20:08:21 +01:00
Igor Velkov
74e8171074 Remove -Wno-error=unknown-warning-option from clang KCFLAGS.
This flag was breaking kernel's cc-option detection, causing GCC-specific
warnings (-Wpacked-not-aligned, -Wstringop-truncation, -Wmaybe-uninitialized)
to be incorrectly added to btrfs/drm/coresight builds when using clang.
2026-01-08 12:04:39 +01:00
Igor Pecovnik
4d60ce08f2 chore: update copyright years to 2026
Update all copyright notices in shell scripts from 2025 to 2026.

## Changes

- **Igor Pecovnik**: 2013-2025 → 2013-2026 (129 files)
- **Ricardo Pardini**: 2023-2025 → 2023-2026, 2020-2025 → 2020-2026 (5 files)

## Additional Improvements

Also updated the backtitle in `lib/functions/configuration/interactive.sh`:
- Changed title from "Armbian building script" to "Armbian Linux build framework"
- Removed docs link for cleaner display
- Uses dynamic year calculation with separate declaration (fixes shellcheck SC2155)
2025-12-25 12:03:34 +01:00
Rolf Leggewie
e624fd805f lib / kernel-make.sh: drop undefined make_filter call. Closes #8529 2025-08-27 13:46:16 +08:00
Ricardo Pardini
6d9a06bf72 shellfmt: lib/functions; no changes 2024-11-25 11:32:28 +01:00
ColorfulRhino
6e6fbac503 cli: kernel-make: Use build@armbian instead of armbian@next for KBUILD 2024-08-15 20:35:26 +00:00
ColorfulRhino
e8c4bde80f cli: kernel-make: Add note for parallel compression for xz
Source: https://lore.kernel.org/lkml/20200901151002.988547791@linuxfoundation.org/
2024-08-15 20:35:26 +00:00
Evgenii Zheltonozhskii
0aa2371db5 Support kernel compilation with Clang/LLVM 2024-08-15 13:20:01 +02:00
ColorfulRhino
d75b1816ff cli: Improve formatting on dts-check scripts with shellfmt.sh 2024-06-25 18:11:43 +02:00
ColorfulRhino
de81f10b0d cli: Add command "dts-check"
Validates the dts/dtb file for the selected board and outputs the validation logs to the user.
This can be used when adding a new board, developing or improving a dts file. Should lead to higher quality device trees and patches overall, if used.
Will show warnings/errors if patches patch in some functionalities to a devicetree file without patching in the dt-bindings .yaml at the same time.
2024-06-25 07:06:21 +02:00
Igor
aa9fea7797 Revert "cli: Add command "dts-check""
This reverts commit 54618c1bd3.
2024-04-26 20:46:42 +08:00
ColorfulRhino
54618c1bd3 cli: Add command "dts-check"
Validates the dts/dtb file for the selected board and outputs the validation logs to the user.
This can be used when adding a new board, developing or improving a dts file. Should lead to higher quality device trees and patches overall, if used.
2024-04-26 18:18:56 +08:00
Darsey Litzenberger
ce33bddb58 Make the colorized build output readable on a light background
This fixes the readability of colorized output of compile.sh when
running on a terminal with a light background.  It uses the COLORFGBG
environment variable similarly to how the ip(8) command does.

Signed-off-by: Darsey Litzenberger <dlitz@dlitz.net>
2024-02-09 19:38:03 +01:00
Ricardo Pardini
11df81403d config/lib/exts: run shellfmt
- `bash lib/tools/shellfmt.sh`
- mostly spacing changes that accumulated over time
2023-08-22 14:28:14 +02:00
amazingfate
ecac08cc8e kernel image files use standard flavor 2023-08-11 17:06:38 +08:00
Ricardo Pardini
e74ab59f48 kernel-make: introduce KERNEL_EXTRA_CFLAGS; fill it to add to what's passed to KCFLAGS; related to #5117 2023-05-01 22:46:43 +02:00
Ricardo Pardini
321a0cd257 revert: run kernel make through pipetty for _moar colors_ (breaks make menuconfig) 2023-04-18 09:27:38 +02:00
Ricardo Pardini
9218401cd8 kernel: fix for 6.3+ linux-headers pkg due to upstream dir change
- extra: run kernel make through `pipetty` for _moar colors_
2023-04-15 22:44:43 +02:00
Igor Pečovnik
b7b8eb7b72
Add / modify (c) in bash scripts (#4922)
* Add / modify (c) in bash scripts

Signed-off-by: Igor <igor@armbian.com>

* Add (c) to the source config files

---------

Signed-off-by: Igor <igor@armbian.com>
2023-03-09 18:30:40 +01:00
Ricardo Pardini
78dcf01c92
armbian-next: cleanup, kill and add to-do's, remove dead code, turn down logging, squash future shortcircuit bugs
- killed `[[ $ROOTFS_TYPE != ext4 ]] && display_alert "Assuming ${BOARD} ${BRANCH} kernel supports ${ROOTFS_TYPE}" "" "wrn"`
  - which definitely didn't belong in rootfs
- disable usage of run_host_command_logged_long_running for kernel-make
- stop lying about long_running stuff being any different from non-long_running versions
- stop lying about 'set -e' when there's still a bunch of pipes for "pv" and stuff all around
2023-02-18 07:44:33 -03:00
Ricardo Pardini
2d9f9216eb
armbian-next: introduce tmpfs-utils.sh; put LOGDIR and WORKDIR under tmpfs; set CCACHE_TEMPDIR under WORKDIR
- introduce generic `prepare_tmpfs_for()`, which manages it's own cleanup/dir removal
- use it for `WORKDIR` (which is `TMPDIR`) and `LOGDIR`
  - adapt previous cleanup handlers for those, so they delete their contents but not the dir itself (which might be mounted)
- also: make `ARMBIAN_LOG_CLI_ID` readonly together with other superglobals
- set `XDG_RUNTIME_DIR` & `XDG_RUNTIME_DIR` together with `TMPDIR`
- kernel-make.sh: pass `CCACHE_TEMPDIR` down to Kernel make (thanks @the-Going)
2023-02-18 07:43:09 -03:00
Ricardo Pardini
53266feff8
armbian-next: kernel: deterministic .config mtime handling; kernel build logging sections reorg
- .config is now compared with previous one byte-wise, diff shown, and mtime preserved directly via `cp -p`
- this does away with the previous mtime-based BS I had written during the fasthash era (and thus remove file-mtime.sh, regen lib)
- split some functions, to better control the interactive piece in the _middle_ of kernel config process
- no more `kernel_config_maybe_interactive()`, now `kernel_config()` that controls its own logging sections
- extract `kernel_determine_toolchain()` out of config into make
- introduce internal hook `armbian_kernel_config()` that runs before custom version (for .config defaults, coming soon)
2023-02-18 07:42:39 -03:00
Ricardo Pardini
76e276c6a9
armbian-next: Python patching delusion, pt1 & pt2 & pt3
- WiP: Python patching delusion, pt 1: finding & parsing patches; apply & git commit with pygit2; Markdown summaries (also for aggregation); git-to-patches tool
  - Python: Markdown aggregation and patching summaries; collapsible; SummarizedMarkdownWriter
  - Python: Markdown aggregation and patching summaries
  - Python: reorg a bit into common/armbian_utils; define the `ASSET_LOG_BASE` in preparation for Markdown delusion
  - Python patching: initial apply patches & initial commit patches to git (using pygit2)
  - Python patching: add basic `series.conf` support
  - Python patching: force use of utf-8; better error handling; use realpath of dirs
  - Python patching: `git-to-patches` initial hack. not proud. half-reused some of the patches-to-git
  - Python patching: "tag" the git commits with info for extracting later; introduce REWRITE_PATCHES/rewrite_patches_in_place
  - Python patching: commented-out, recover-bad-patches hacks
  - Python patching: shorten the signature
  - Python patching: allow BASE_GIT_TAG as well as BASE_GIT_REVISION
  - Python patching: git-archeology for patches missing descriptions; avoid UTF-8 in header/desc (not diff)
  - Python patching: use modern-er email.utils.parsedate_to_datetime to parse commit date
  - Python patching: unify PatchInPatchFile; better git-commiting; re-exporting patches from Git (directly)
  - Python patching: switch to GitPython
    - GitPython is like 100x slower than pygit2, but actually allows for date & committer
    - also allows to remove untracked files before starting
  - Python aggregation: fix missing `AGGREGATED_APT_SOURCES_DICT`
  - Python patching: add `unidecode` dependency to pip3 install
  - Python patching: don't try archeology if SRC is not a Git Repo (eg, in Docker)
  - Python patching: don't try archeology if not applying patches to git
- WiP: Python patching delusion, pt2: actually use for u-boot & kernel patching
  - Python patching: much better problem handling/logging; lenient with recreations (kernel)
  - Python patching: don't force SHOW_LOG for u-boot patching
  - Python patching: don't bomb for no reason when there are no patches to apply
  - Python patching: fully (?) switch kernel patching to Python
  - Python patching: more logging fixups
  - Python patching: capture `kernel_git_revision` from `fetch_from_repo()`'s `checked_out_revision`
  - Python patching: fully switch u-boot patching to Python
  - Python aggregation/patching: colored logging; patching: always reset to git revision
  - Python aggregation/patching: better logging; introduce u-boot Python patching
- Python patching pt3: recovers and better Markdown
  - Python patching: detect, and rescue, `wrong_strip_level` problem; don't try to export patches that didn't apply, bitch instead
  - Python patching: Markdown patching summary table, complete with emoji
  - Python patching: include the problem breakdown in Markdown summary
  - Python patching: sanity check against half-bare, half-mbox patches
  - Python patching: try to recover from 1) bad utf-8 encoded patches; 2) bad unidiff patches; add a few sanity checks
  - Python patching: try, and fail, to apply badly utf-8 encoded patches directly as bytes [reverted]
  - Python patching: try to recover from patch *parse* failures; show summary; better logging
      - set `GIT_ARCHEOLOGY=yes` to do archeology, default not

- armbian-next: Python `pip` dependencies handling, similar to `hostdeps`
  - same scheme for Dockerfile caching
  - @TODO: still using global/shared environment; should move to a dir under `cache` or some kinda venv
- WiP: add `python3-pip` to hostdeps; remove `python-setuptools`
  - remove `python-setuptools` (Python2, no longer exists in Sid) from hostdeps
  - add `python3-pip` to hostdeps; part of virtualenv saga
- WiP: split `kernel.sh` a bit, into `kernel-patching.sh`, `kernel-config.sh` and `kernel-make.sh`
  - `advanced_patch()`: rename vars for clarity; no real changes
- Python patching: introduce FAST_ARCHEOLOGY; still trying for Markdown links
2023-02-18 07:40:52 -03:00