Commit Graph

89 Commits

Author SHA1 Message Date
Igor Velkov
1b74748622
Extension: ccache-remote — shared compilation cache via Redis/HTTP (#9369)
* add hook to allow customizing before kernel make env creation
* Hook runs in docker_cli_prepare_launch() just before DOCKER_EXTRA_ARGS
is processed, allowing extensions to add Docker arguments with a more
descriptive hook name than add_host_dependencies.
* Extension: ccache-remote

Enables ccache with remote Redis storage for sharing compilation cache across build hosts.

Features:
- Auto-discovery via Avahi/mDNS (ccache.local hostname)
- Explicit Redis server configuration via CCACHE_REMOTE_STORAGE
- Build statistics display at end of build (hit/miss/error rates)
- Support for both Docker and native builds
- Hooks for kernel and u-boot compilation environments

Documentation includes server setup instructions with security warnings,
client mDNS configuration, and cache sharing requirements.


* uboot: fix ccache environment and add extension hook

U-Boot build uses `env -i` which clears all environment variables.
CCACHE_DIR and CCACHE_TEMPDIR were not explicitly passed to make,
unlike kernel build (kernel-make.sh). This caused ccache to use
default directory instead of configured Armbian one, breaking
cache statistics and shared cache functionality.

Changes:
- Add CCACHE_DIR and CCACHE_TEMPDIR to uboot_make_envs
- Add uboot_make_config hook for extensions (similar to kernel_make_config),
  allowing modification of environment variables before compilation

* add long list of allowed ccache-related env vars
* set permissions to ccache files RW for everyone if cache not private
* ccache: add ccache_post_compilation hook for extensions
* ccache-remote: use ccache_post_compilation hook instead of cleanup handler

Show remote ccache stats after each compilation (kernel, uboot) via hook,
instead of once at the end via cleanup handler. Stats now shown even on
build failure.

* ccache: show stats with safe arithmetic
* ccache/uboot: improve code comments per review feedback

- uboot.sh: clarify ARMBIAN=foe workaround for dual-compiler scenario
- ccache-remote.sh: document that CCACHE_REDIS_CONNECT_TIMEOUT must be
  set before extension loads

* ccache-remote: mask storage URLs in logs

Mask CCACHE_REMOTE_STORAGE when emitting Docker env debug logs.

* ccache-remote: extract ccache_inject_envs() helper to deduplicate passthrough loops

Extract ccache_inject_envs() helper to deduplicate identical passthrough
loops in kernel and uboot make config hooks.

ccache-remote: rename functions to follow project naming conventions

Rename get_redis_stats and mask_storage_url to ccache_get_redis_stats
and ccache_mask_storage_url to follow project naming conventions.

ccache-remote: mask credentials in debug log output for passthrough loops

Mask CCACHE_REMOTE_STORAGE value through ccache_mask_storage_url() before
logging in both Docker env and make env passthrough loops to avoid leaking
credentials into build logs.

* ccache-remote: add HTTP/WebDAV backend and DNS discovery
* ccache-remote: move extension script into directory layout
* ccache-remote: add server setup docs and config files
* ccache-remote: validate Redis credentials in URLs
* ccache-remote: document Redis auth options and safe passwords

Add separate insecure config example for trusted networks.

Recommend URL-safe hex passwords and update setup docs.

* ccache-remote: improve Docker loopback handling and IPv6 host parsing
2026-03-01 01:18:35 +01:00
Igor Pecovnik
4c963e2930 docker: improve auto-pull cronjob with opt-in flag and cleanup
- Add ARMBIAN_DOCKER_AUTO_PULL environment variable (opt-in, must be explicitly set to "yes")
- Move auto-pull cronjob setup from requirements to docker CLI
- Add automatic cleanup of cronjob files when flag is disabled/removed
- Remove verbose "unchanged" messages for cleaner output
- Simplify control flow in docker_ensure_auto_pull_cronjob()
2026-02-01 00:22:54 +01:00
Igor Pecovnik
882d7e3dfd docker: add automatic image pull cronjob and cleanup system
- Add docker_cleanup_old_images() to remove dangling images and keep only 2 most recent per tag
- Add docker_pull_with_marker() to pull images and update marker files tracking last pull time
- Add docker_setup_auto_pull_cronjob() to create/update system cronjob and wrapper script via hash-based detection
- Add docker_ensure_auto_pull_cronjob() to ensure cronjob is installed and up-to-date
- Create self-contained wrapper script at /usr/local/bin/armbian-docker-pull for cron execution
- Store configuration hash in /var/lib/armbian/docker-pull.hash for smart update detection
- Install cronjob at /etc/cron.d/armbian-docker-pull to pull images every 12 hours
- Move cronjob setup from docker_cli_prepare() to requirements command
- Cronjob is now only installed when users explicitly run ./compile.sh requirements
- Prevents "12 hours since last pull, pulling again" delay during builds

Signed-off-by: Igor Pecovnik <igor@armbian.com>
2026-02-01 00:22:54 +01:00
Igor Velkov
3e71438048 Fix locale warnings during chroot operations
Build process generated numerous locale warnings like:
bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
perl: warning: Setting locale failed

This happened because the Docker container and host environment were
configured to use `en_US.UTF-8`, but the rootfs cache may not contain
this locale (only `C.utf8` and whatever `DEST_LANG` specifies,
e.g. `en_GB.utf8`).

When `chroot_sdcard` runs commands inside the rootfs, environment variables
are inherited from the host/Docker, causing locale lookup failures.

Then use `C.UTF-8` locale for the build environment instead of `en_US.UTF-8`.
This locale is always available in rootfs immediately after mmdebstrap,
requiring no generation.

Changes
`lib/functions/host/docker.sh`: Remove `en_US.UTF-8` locale generation,
set `LANG=C.UTF-8` in container environment
`lib/functions/host/prepare-host.sh`: Change locale exports
from `en_US.UTF-8` to `C.UTF-8`
2026-01-08 12:24:18 +01:00
Igor Pecovnik
4d60ce08f2 chore: update copyright years to 2026
Update all copyright notices in shell scripts from 2025 to 2026.

## Changes

- **Igor Pecovnik**: 2013-2025 → 2013-2026 (129 files)
- **Ricardo Pardini**: 2023-2025 → 2023-2026, 2020-2025 → 2020-2026 (5 files)

## Additional Improvements

Also updated the backtitle in `lib/functions/configuration/interactive.sh`:
- Changed title from "Armbian building script" to "Armbian Linux build framework"
- Removed docs link for cleaner display
- Uses dynamic year calculation with separate declaration (fixes shellcheck SC2155)
2025-12-25 12:03:34 +01:00
Jesse R Codling
d1fc474d50 Typo fix from @tabrisnet
Co-authored-by: tabrisnet <96966042+tabrisnet@users.noreply.github.com>
2025-12-07 08:49:22 +01:00
Jesse R Codling
9f164a21ce Use docker --version for podman detection 2025-12-07 08:49:22 +01:00
Jesse R Codling
3fb2606d00 Conditionally add volume flags for building with Podman
Use $DOCKER_INFO to check for podman vs dockerd, and conditionally
`exec` and `dev` volume options, which are needed on podman but not
allowed on dockerd.
2025-12-07 08:49:22 +01:00
Jesse R Codling
3aea2de999 Add needed volume mount opts for Docker/Podman
It hasn't been an issue for Docker because their defaults didn't
conflict, but Podman mounts volumes with nodev (and formerly noexec) by
default, which would break the build. Adding `dev` and `exec` to the
volume mount options ensures the needed setting whatever the defaults.
2025-12-07 08:49:22 +01:00
Rolf Leggewie
bda921595b minor spelling mistakes 2025-08-02 19:23:38 +08:00
Werner
15407330c6
Preparation for http proxy caching (#8281)
* always use http for apt repos

* pass possible proxy arguments to docker

* populate lower-case env variables with upper-case ones if not set otherwise
2025-06-13 10:31:24 +02:00
Ricardo Pardini
03c47689cb docker: 50%+ faster during "enabling all extensions looking for Docker dependencies"
- 7s+ -> 3s
- by using `enable_extensions_with_hostdeps_builtin_and_user "add_host_dependencies" "host_dependencies_known"` instead of `enable_all_extensions_builtin_and_user`
2025-05-29 17:58:28 +02:00
Igor Pecovnik
4dc9b00fb0 Framework: switch Docker default image to Noble 2025-05-14 11:59:12 +02:00
JohnTheCoolingFan
bed3f85213 Run docker via run_host_command_logged 2024-08-18 10:40:32 +02:00
JohnTheCoolingFan
1045fda3d4 Check for loop devices and a temp container run 2024-08-18 10:40:32 +02:00
Viacheslav Bocharov
7875fce1e9 Allow to pass docker login credentials to oras-cli for and CI environment
Signed-off-by: Viacheslav Bocharov <adeep@lexina.in>
2024-07-04 21:35:26 +02:00
Igor Pecovnik
024c0866ad DTS-check: Python librarires have troubles to build, disabling for now
- Add requirements.txt to Dockerfile generation
2024-06-25 08:53:22 +02:00
Darsey Litzenberger
ce33bddb58 Make the colorized build output readable on a light background
This fixes the readability of colorized output of compile.sh when
running on a terminal with a light background.  It uses the COLORFGBG
environment variable similarly to how the ip(8) command does.

Signed-off-by: Darsey Litzenberger <dlitz@dlitz.net>
2024-02-09 19:38:03 +01:00
Gunjan Gupta
7b81255a5f Fix compilation within docker-shell 2024-02-01 21:04:23 +05:30
Ricardo Pardini
2b6a90484a docker launcher: don't lose track of the wanted host_release when setting using Armbian base image
- host_release (wanted_release_tag) was being re-computed from  DOCKER_ARMBIAN_BASE_IMAGE
- DOCKER_ARMBIAN_BASE_IMAGE might have been changed from "<os>:<release>" if DOCKERFILE_USE_ARMBIAN_IMAGE_AS_BASE != no
- that would cause host_release to be "armbian-ubuntu-lunar-latest", not "lunar"
  - which would cause python2/python2-dev to be installed, which would cause Dockerfile build to fail
2023-10-20 12:06:32 +00:00
Ricardo Pardini
19226a9c27 docker (launcher): fix DOCKER_PASS_GIT=yes to actually work
- it still sucks (very slow) but is a way to get ${SRC}/.git in Docker if needed
  - eg for patching archeology
2023-10-03 14:11:42 +02:00
Ricardo Pardini
cacd3c9b08 docker (launcher): fix un-needed grep escape \: (fixes "grep: warning: stray \ before :" on recent coreutils) 2023-10-03 14:11:42 +02:00
Gunjan Gupta
1d6a779ed1 Drop HOST OS that doesnot support crust compilation 2023-08-22 13:03:51 +02:00
Ricardo Pardini
64b5220853
debian: add Trixie (Debian 13, testing) csc target & build host (#5621)
- Trixie, like Bookworm and Sid, does not have python2 anymore
2023-08-22 08:14:55 +02:00
Ricardo Pardini
e6f70a3573 docker: default COLUMNS env var to 160 if unset / not-interactive 2023-07-01 14:55:49 +02:00
Igor
1848366170 Revert "docker: use ${SRC} as DOCKER_ARMBIAN_TARGET_PATH instead of hardcoded /armbian"
This reverts commit 6dd7067c79.
2023-06-13 21:59:44 +02:00
Ricardo Pardini
6dd7067c79 docker: use ${SRC} as DOCKER_ARMBIAN_TARGET_PATH instead of hardcoded /armbian
- this way host-side and Docker-side `SRC` are now the same path
  - messages containing the path to a file will now match across Docker and actual host
2023-06-13 19:32:49 +02:00
Ricardo Pardini
70a64a0415 patching: Patching Summary and Patching Failures tables using Python's Rich
- even Rich'er patch output by colorizing certain strings green/yellow/red
- BASE_GIT_TAG now very sneakily also accepts a branch name
- IMPORTANT: this includes: BREAKING CHANGE: patches failing to apply now break the build. fixes #4958
  - also break on legacy `process_patch_file()` failure, remove `EXIT_PATCHING_ERROR`
2023-06-13 12:29:52 +02:00
Ricardo Pardini
52710b3363 git/docker: bsp-cli: fix BUILD_REPOSITORY_URL and BUILD_REPOSITORY_COMMIT under Docker; fixes AR-1750 2023-05-24 19:20:57 +02:00
Ricardo Pardini
234232085d docker: remove dead code 2023-05-16 20:09:47 +02:00
Ricardo Pardini
f0f9159faa logging: curb CI/GHA special errors for cascading errors during build failure
- otherwise we get 4 errors for each "error" in CI
  - one for the real error
  - one for "wait for cleanups"
  - one for "Docker build failed"
  - one from GHA itself since command failed (we can't get rid of this)
2023-05-15 17:43:25 +02:00
Ricardo Pardini
73e9c89807 docker: fix: always mount .docker/config.json if it exists (in GHA) 2023-05-03 15:09:56 +02:00
Pander
0d72163868 Fixed execution rights docker.sh 2023-04-22 20:08:26 +02:00
Ricardo Pardini
d0716b65c0 docker/sudo: better control/info over CLI relaunch args; introduce CLI relaunch envs
- allow adding re-launch environments by populating `ARMBIAN_CLI_RELAUNCH_PARAMS` dict
- similar to `ARMBIAN_CLI_RELAUNCH_PARAMS`
- rename/consolidate `()`'s output globals:
  - `ARMBIAN_CLI_FINAL_RELAUNCH_ARGS` (rename)
  - `ARMBIAN_CLI_FINAL_RELAUNCH_ENVS` (new)
    - includes `ARMBIAN_HIDE_REPEAT_PARAMS` automatically
- `sudo`: include envs when relaunching, use bash explicitly
- `docker`: better logging
- `docker`: include envs when relaunching via `--env` args
2023-04-21 14:26:33 +02:00
Ricardo Pardini
ed0b406265 lib: fix: replace undue export statements with declare -g 2023-03-31 09:12:44 +02:00
Ricardo Pardini
fe7a24885b docker: introduce DOCKER_SKIP_UPDATE=yes to skip updating prebuilt Docker image, only when using 2023-03-31 09:12:44 +02:00
Ricardo Pardini
40c5df55ab docker: introduce ${DOCKER_ARMBIAN_BASE_COORDINATE_PREFIX:-"ghcr.io/armbian/docker-armbian-build:armbian-"} for easy override 2023-03-31 09:12:44 +02:00
Ricardo Pardini
9931350483 docker: fix: pass down OCI_TARGET_BASE to Docker instance automatically 2023-03-17 08:32:11 +01:00
Michael Beck
1223b22ffd
WSL2: Ensure the Windows Terminal is detected under WSL2 docker build. (#4915)
- by passing the WT_SESSION environment variable into the docker container  if it is set
2023-03-09 19:19:30 +01:00
Igor Pečovnik
b7b8eb7b72
Add / modify (c) in bash scripts (#4922)
* Add / modify (c) in bash scripts

Signed-off-by: Igor <igor@armbian.com>

* Add (c) to the source config files

---------

Signed-off-by: Igor <igor@armbian.com>
2023-03-09 18:30:40 +01:00
Ricardo Pardini
aa70ad570b
armbian-next: use new/definitive pkg/image coordinates for Docker image, on Armbian org 2023-02-18 12:13:42 -03:00
Ricardo Pardini
20de5dd2c6
armbian-next: switch to using Armbian-org GH hosted git shallow trees and Docker images 2023-02-18 07:46:46 -03:00
Ricardo Pardini
e697f2e67c
armbian-next: docker: curb logging about OCI_TARGET_BASE 2023-02-18 07:46:10 -03:00
Ricardo Pardini
9bffa5e749
armbian-next: artifacts: firmware/rootfs and actual usage of artifacts during image build
- artifacts: introduce `ARTIFACT_IGNORE_CACHE=yes`
- artifacts: introduce `DONT_BUILD_ARTIFACTS`, list of artifacts that if not found cached, fail the build
- kernel_package_source() is no more
- a long dissertation about kernels, families, and the universe
- artifacts: actually use rootfs artifact for image build
- artifacts: detangle via artifact_base_dir
- artifacts: rootfs: use folders in artifact_name; include cache_type
- artifacts: some cleanups / validations
- rootfs artifact; drop old rootfs cli
- artifacts: new CLI shortcuts; remove old firmware CLI
- artifacts: full firmware & usage
- use firmware artifacts in image build and install its debs
- kernel artifact sans legacy; use tmpdir for .deb target for all packages
- legacy artifact versions is no more; pack/unpack now in common obtain;
- artifacts: uboot: cleanup legacy renaming, use artifact version directly
- artifacts: add firmware (small) artifact
- deploy uboot to loop from artifact; allow tty to artifact; todos for cleaning
- fixes, kernel dtb/headers conditional; remove `.git` from Makefile url; use mapfile for finding files to hash
  - completely remove KERNEL_HAS_WORKING_HEADERS_FULL_SOURCE and `kernel_package_callback_linux_headers_full_source()`
  - don't use realpath for artifact_file_relative
  - curb some warnings
  - fix: only install headers & dtbs if such exist
- kernel .config hook modification hash versioning
- OCI_TARGET_BASE vs per-artifact defaults; only deploy to remote from CLI with OTB
- artifact revolver & installing into image
  - add artifact_map_packages and artifact_map_debs dicts
  - revolver accumulates all info
  - REPOSITORY_INSTALL is no more (for uboot/kernel, later others)
  - rename `VER` to `IMAGE_INSTALLED_KERNEL_VERSION`
2023-02-18 07:46:03 -03:00
Ricardo Pardini
a75e58c320
armbian-next: docker: show $SECONDS in Docker dash logs 2023-02-18 07:46:01 -03:00
Ricardo Pardini
ca6bf9046b
armbian-next: remove all traces of Vagrant and FEL boot 2023-02-18 07:46:00 -03:00
Ricardo Pardini
399d9964dd
armbian-next: python/pip: include pip pkgs in Dockerfile; use them if available instead of downloading 2023-02-18 07:45:51 -03:00
Ricardo Pardini
0e480e685e
armbian-next: docker: pass down git info via env var, since Docker doesn't get ${SRC}/.git, yet we need that info in the Docker logs 2023-02-18 07:45:40 -03:00
Ricardo Pardini
7431984025
armbian-next: docker: pass CARD_DEVICE to Docker via --device 2023-02-18 07:45:35 -03:00
Ricardo Pardini
956ef9ac7d
armbian-next: fix: for the 1000th time had broken KERNEL_ONLY=yes; curb some warnings 2023-02-18 07:45:18 -03:00