armbian-build/lib/functions/compilation/ccache.sh
Igor Velkov 1b74748622
Extension: ccache-remote — shared compilation cache via Redis/HTTP (#9369)
* add hook to allow customizing before kernel make env creation
* Hook runs in docker_cli_prepare_launch() just before DOCKER_EXTRA_ARGS
is processed, allowing extensions to add Docker arguments with a more
descriptive hook name than add_host_dependencies.
* Extension: ccache-remote

Enables ccache with remote Redis storage for sharing compilation cache across build hosts.

Features:
- Auto-discovery via Avahi/mDNS (ccache.local hostname)
- Explicit Redis server configuration via CCACHE_REMOTE_STORAGE
- Build statistics display at end of build (hit/miss/error rates)
- Support for both Docker and native builds
- Hooks for kernel and u-boot compilation environments

Documentation includes server setup instructions with security warnings,
client mDNS configuration, and cache sharing requirements.


* uboot: fix ccache environment and add extension hook

U-Boot build uses `env -i` which clears all environment variables.
CCACHE_DIR and CCACHE_TEMPDIR were not explicitly passed to make,
unlike kernel build (kernel-make.sh). This caused ccache to use
default directory instead of configured Armbian one, breaking
cache statistics and shared cache functionality.

Changes:
- Add CCACHE_DIR and CCACHE_TEMPDIR to uboot_make_envs
- Add uboot_make_config hook for extensions (similar to kernel_make_config),
  allowing modification of environment variables before compilation

* add long list of allowed ccache-related env vars
* set permissions to ccache files RW for everyone if cache not private
* ccache: add ccache_post_compilation hook for extensions
* ccache-remote: use ccache_post_compilation hook instead of cleanup handler

Show remote ccache stats after each compilation (kernel, uboot) via hook,
instead of once at the end via cleanup handler. Stats now shown even on
build failure.

* ccache: show stats with safe arithmetic
* ccache/uboot: improve code comments per review feedback

- uboot.sh: clarify ARMBIAN=foe workaround for dual-compiler scenario
- ccache-remote.sh: document that CCACHE_REDIS_CONNECT_TIMEOUT must be
  set before extension loads

* ccache-remote: mask storage URLs in logs

Mask CCACHE_REMOTE_STORAGE when emitting Docker env debug logs.

* ccache-remote: extract ccache_inject_envs() helper to deduplicate passthrough loops

Extract ccache_inject_envs() helper to deduplicate identical passthrough
loops in kernel and uboot make config hooks.

ccache-remote: rename functions to follow project naming conventions

Rename get_redis_stats and mask_storage_url to ccache_get_redis_stats
and ccache_mask_storage_url to follow project naming conventions.

ccache-remote: mask credentials in debug log output for passthrough loops

Mask CCACHE_REMOTE_STORAGE value through ccache_mask_storage_url() before
logging in both Docker env and make env passthrough loops to avoid leaking
credentials into build logs.

* ccache-remote: add HTTP/WebDAV backend and DNS discovery
* ccache-remote: move extension script into directory layout
* ccache-remote: add server setup docs and config files
* ccache-remote: validate Redis credentials in URLs
* ccache-remote: document Redis auth options and safe passwords

Add separate insecure config example for trusted networks.

Recommend URL-safe hex passwords and update setup docs.

* ccache-remote: improve Docker loopback handling and IPv6 host parsing
2026-03-01 01:18:35 +01:00

111 lines
4.0 KiB
Bash

#!/usr/bin/env bash
#
# SPDX-License-Identifier: GPL-2.0
#
# Copyright (c) 2013-2026 Igor Pecovnik, igor@armbian.com
#
# This file is a part of the Armbian Build Framework
# https://github.com/armbian/build/
# Parse a single numeric field from "ccache --print-stats" tab-separated output
# Returns 0 if field not found or not numeric
function ccache_get_stat() {
local stats_output="$1" field="$2"
local val
val=$(echo "$stats_output" | grep "^${field}" | cut -f2 || true)
[[ "${val}" =~ ^[0-9]+$ ]] || val=0
echo "$val"
}
# Calculate hit percentage from hit and miss counts
function ccache_hit_pct() {
local hit="$1" miss="$2"
local total=$(( hit + miss ))
if [[ $total -gt 0 ]]; then
echo $(( hit * 100 / total ))
else
echo 0
fi
}
# Helper function to show ccache stats - used as cleanup handler for interruption case
function ccache_show_compilation_stats() {
local stats_output direct_hit direct_miss pct
stats_output=$(ccache --print-stats 2>&1 || true)
direct_hit=$(ccache_get_stat "$stats_output" "direct_cache_hit")
direct_miss=$(ccache_get_stat "$stats_output" "direct_cache_miss")
pct=$(ccache_hit_pct "$direct_hit" "$direct_miss")
display_alert "Ccache result" "hit=${direct_hit} miss=${direct_miss} (${pct}%)" "info"
# Hook for extensions to show additional stats (e.g., remote storage)
call_extension_method "ccache_post_compilation" <<- 'CCACHE_POST_COMPILATION'
*called after ccache-wrapped compilation completes (success or failure)*
Useful for displaying remote cache statistics or other post-build info.
CCACHE_POST_COMPILATION
}
function do_with_ccache_statistics() {
display_alert "Clearing ccache statistics" "ccache" "ccache"
run_host_command_logged ccache --zero-stats
if [[ "${SHOW_CCACHE}" == "yes" ]]; then
# show value of CCACHE_DIR
display_alert "CCACHE_DIR" "${CCACHE_DIR:-"unset"}" "ccache"
display_alert "CCACHE_TEMPDIR" "${CCACHE_TEMPDIR:-"unset"}" "ccache"
# determine what is the actual ccache_dir in use
local ccache_dir_actual
ccache_dir_actual="$(ccache --show-config | grep "cache_dir =" | cut -d "=" -f 2 | xargs echo)"
# calculate the size of that dir, in bytes.
local ccache_dir_size_before ccache_dir_size_after ccache_dir_size_before_human
ccache_dir_size_before="$(du -sb "${ccache_dir_actual}" | cut -f 1)"
ccache_dir_size_before_human="$(numfmt --to=iec-i --suffix=B --format="%.2f" "${ccache_dir_size_before}")"
# show the human-readable size of that dir, before we start.
display_alert "ccache dir size before" "${ccache_dir_size_before_human}" "ccache"
# Show the ccache configuration
wait_for_disk_sync "before ccache config"
display_alert "ccache configuration" "ccache" "ccache"
run_host_command_logged ccache --show-config "&&" sync
fi
# Register cleanup handler to show stats even if build is interrupted
add_cleanup_handler ccache_show_compilation_stats
display_alert "Running ccache'd build..." "ccache" "ccache"
local build_exit_code=0
"$@" || build_exit_code=$?
# Show stats and remove from cleanup handlers (so it doesn't run twice on exit)
execute_and_remove_cleanup_handler ccache_show_compilation_stats
# Re-raise the error if the build failed
if [[ ${build_exit_code} -ne 0 ]]; then
return ${build_exit_code}
fi
if [[ "${SHOW_CCACHE}" == "yes" ]]; then
display_alert "Display ccache statistics" "ccache" "ccache"
run_host_command_logged ccache --show-stats --verbose
# calculate the size of that dir, in bytes, after the compilation.
ccache_dir_size_after="$(du -sb "${ccache_dir_actual}" | cut -f 1)"
# calculate the difference, in bytes.
local ccache_dir_size_diff
ccache_dir_size_diff="$((ccache_dir_size_after - ccache_dir_size_before))"
# calculate the difference, in human-readable format; numfmt is from coreutils.
local ccache_dir_size_diff_human
ccache_dir_size_diff_human="$(numfmt --to=iec-i --suffix=B --format="%.2f" "${ccache_dir_size_diff}")"
# display the difference
display_alert "ccache dir size change" "${ccache_dir_size_diff_human}" "ccache"
fi
return 0
}