Commit Graph

163 Commits

Author SHA1 Message Date
Ricardo Pardini
b2d02071bd
armbian-next: better ccache logs/stats/Docker-support/use in u-boot/kernel
- introduce `SHOW_CCACHE=yes` for detailed ccache statistics and logging
- fix: kernel build is done under "env -i", so pass CCACHE_DIR down to Make if CCACHE_DIR is set...
- split from kernel.sh, show stats also for u-boot targets; show compile time
- add volume definition (under `${SRC}/cache/ccache`); auto-use that in Docker builds via `CCACHE_DIR`
- better logging for `do_with_ccache_statistics()`
- there's some CCACHE_DIR code from before; unify @TODO
2023-02-18 07:40:19 -03:00
Ricardo Pardini
093336e914
armbian-next: cli: distccd launcher for build farm
- relaunched under docker
- possibly much nicer if zeroconf enabled?
- this thing is ignoring signals. can't ctrl-c out of it. why? @TODO
- pt 1, subpt 3: distccd - distcc-pump - avahi
- pt 1, subpt 3: distccd + distcc-pump
- pt 1, subpt 2: distccd + avahi
- pt 1, subpt 1: distccd
2023-02-18 07:40:18 -03:00
Ricardo Pardini
da0b3080a0
armbian-next: distcc: optionally using distcc for kernel build 2023-02-18 07:40:17 -03:00
Ricardo Pardini
81376df4b7
armbian-next: u-boot: use scripts/config instead of seding into the .config (BOOTDELAY, CONFIG_LOGLEVEL=6, etc); v2022.10+ only 2023-02-18 07:40:16 -03:00
Ricardo Pardini
7939257592
armbian-next: git: do not delete working copies on remote origin mismatch; accept empty origins as valid 2023-02-18 07:40:15 -03:00
Ricardo Pardini
cc62d4932a
armbian-next: introduce ARMBIAN_FIRMWARE_GIT_SOURCE and _BRANCH to allow custom firmware repository coordinates 2023-02-18 07:40:14 -03:00
Ricardo Pardini
93f0545519
armbian-next: the great aggregation rewrite: pt 0: SHOW_AGGREGATION debug logging 2023-02-18 07:40:13 -03:00
Ricardo Pardini
9fbbe897cd
armbian-next: docker: disable showing Docker volume usage; fix syntax in test for ARMBIAN_RUNNING_IN_CONTAINER 2023-02-18 07:40:12 -03:00
Ricardo Pardini
b703eb6da3
armbian-next: docker: locale-gen + enable all existing extensions before calculating docker hostdeps
- that way all extensions' hostdeps are included in Dockerfile
- also run locale-gen in Dockerfile
2023-02-18 07:40:11 -03:00
Ricardo Pardini
94d451b812
armbian-next: introduce hook post_family_config_branch_<branch>() as convenience after post_family_config() hook 2023-02-18 07:40:10 -03:00
Ricardo Pardini
a9e88f58e5
armbian-next: json-info command, wrapping up the Python tooling; running in Docker; fix config dump 2023-02-18 07:40:09 -03:00
Ricardo Pardini
291b02577a
armbian-next: rename compile_plymouth-theme-armbian() to compile_plymouth_theme_armbian() for consistency 2023-02-18 07:40:08 -03:00
Ricardo Pardini
75de1c2d82
armbian-next: remove usage of REQUIREMENTS_DEFS_ONLY; initialize extensions for requirements cli command & docker pre-game deps 2023-02-18 07:40:07 -03:00
Ricardo Pardini
bc7d388b49
armbian-next: cli: a bit more untangling of re-launching; introduce ARMBIAN_CLI_RELAUNCH_COMMAND and cli_standard_relaunch_docker_or_sudo()
- `configdump` cli: force root/docker
2023-02-18 07:40:06 -03:00
Ricardo Pardini
497a3a80ab
armbian-next: cli: de-hardcode build and docker's interdependence, via DOCKER_CLI_CMD
- `undecided` now directly sends to `build`, which might send to `docker`, which relaunches `build` in Docker;
- the _sending to Docker_ part probably needs refactor
2023-02-18 07:40:05 -03:00
Ricardo Pardini
d502402c1a
armbian-next: show more complete gcc versions in logs 2023-02-18 07:40:04 -03:00
Ricardo Pardini
329e249df1
armbian-next: avoid git cramps when deciding to init repos 2023-02-18 07:40:03 -03:00
Ricardo Pardini
749fc349ee
armbian-next: better handling for binfmt_misc woes; don't require it when native build & only complain if it really fails 2023-02-18 07:40:02 -03:00
Ricardo Pardini
884276cf7f
armbian-next: correctly test for systemd is-system-running --quiet 2023-02-18 07:40:01 -03:00
Ricardo Pardini
f061c9e83b
armbian-next: docker: correctly show all Armbian Docker volumes; use bind, not volume, for output/debs 2023-02-18 07:40:00 -03:00
Ricardo Pardini
4ea6519be2
armbian-next: don't uuidgen twice for no reason 2023-02-18 07:39:59 -03:00
Ricardo Pardini
606e82bcfc
armbian-next: docker: handle the case where we can't pull, and don't otherwise have, the base image; in this case use original debian/ubuntu image 2023-02-18 07:39:58 -03:00
Ricardo Pardini
fc3286a3f9
armbian-next: initial post-kinetic resolved debacle, for now just WARN don't break; missing resolved package in kinetic? 2023-02-18 07:39:57 -03:00
Ricardo Pardini
aaa59261bb
armbian-next: mountpoints.sh with all bind/volume directories definitions and looping func; use it to determine best bind/volume combination for each host OS in Docker 2023-02-18 07:39:56 -03:00
Ricardo Pardini
252f96967e
armbian-next: cleanup logging in distro-agnostic.sh; don't use install.log or such 2023-02-18 07:39:55 -03:00
Ricardo Pardini
c4eaf49f3d
armbian-next: cleanup extensions logging, using display_alert new levels extensions (SHOW_EXTENSIONS=yes) and extensionstrace (SHOW_EXTENSIONS_TRACE=yes)
- DEBUG_EXTENSION_CALLS is gone. use SHOW_EXTENSIONS instead.
- LOG_ALL_HOOK_TRACES is gone. use SHOW_EXTENSIONS_TRACE instead
- LOG_ENABLE_EXTENSION is gone. use SHOW_DEBUG=yes instead
2023-02-18 07:39:54 -03:00
Ricardo Pardini
72bc1b09d8
armbian-next: docker: don't even try to to prepare Docker, if it's not available 2023-02-18 07:39:53 -03:00
Ricardo Pardini
3e9c123dd8
armbian-next: docker: try support Rancher Desktop (in dockerd mode), experimental.
- Changes how Docker Desktop works too, there's a `case` for switching back if needed.
- Now the linux-only CONTAINER_COMPAT hacks are used also for Darwin (`DOCKER_SERVER_REQUIRES_LOOP_HACKS=yes`)
  - `/dev/loop*` is spelled out since they don't exist on Darwin (`DOCKER_SERVER_USE_STATIC_LOOPS=yes`)
2023-02-18 07:39:52 -03:00
Ricardo Pardini
3719a059a9
armbian-next: don't force SHOW_LOG=yes during Dockerfile build; we're verbose enough without it 2023-02-18 07:39:51 -03:00
Ricardo Pardini
19e15b8b35
armbian-next: back to including 'cli' sources/apt sources when building non-desktops (eg, nala etc) 2023-02-18 07:39:48 -03:00
Ricardo Pardini
a612a03678
armbian-next: retry 3 times for loop device's size, then give up; introduce RETRY_RUNS in do_with_retries() 2023-02-18 07:39:47 -03:00
Ricardo Pardini
b9daf6d708
armbian-next: extras/buildpkg.sh code is out [deprecated, later completely dropped] 2023-02-18 07:39:46 -03:00
Ricardo Pardini
a54e7b2fae
armbian-next: add gcc-riscv64-linux-gnu to hostdeps (crossbuild-essential-riscv64 is Ubuntu-only) 2023-02-18 07:39:45 -03:00
Ricardo Pardini
bb0e89f017
armbian-next: always include libc6-amd64-cross in hostdeps; qemu is a dummy package now, remove 2023-02-18 07:39:44 -03:00
Ricardo Pardini
d24d3327a8
armbian-next: the great cli entrypoint (+docker) rewrite; introduce USE_LOCAL_APT_DEB_CACHE replacing apt-cacher-ng
- armbian-next: introduce `USE_LOCAL_APT_DEB_CACHE` (default `=yes`) as alternative/in addition to `apt-cacher-ng` (eg, in Docker)
  - this uses `cache/aptcache/${RELEASE}-${ARCH}` (in the host) for
      - apt cache, by bind-mounting it to `${SDCARD}/var/cache/apt` in the `chroot_sdcard_apt_get()` runner and its usages
      - debootstrap, by passing it `--cache-dir`
  - utility function to help understand what is happening to cache during usage
  - apt itself mantains this cache, removing old packages when new ones are installed. apt does this _by default_
      - introduce `DONT_MAINTAIN_APT_CACHE=yes` to skip out of automatic apt maintenance of apt cache, eg, during `remove`s
      - don't do `apt clean` and such if using local cache, that would clean the cache, not the chroot
  - clean up `install_deb_chroot()` a little, find an unrelated bug there
- WiP: the great cli entrypoint (+docker) rewrite, Phase 6: relaunching structure; re-pass ARMBIAN_BUILD_UUID; use ARMBIAN_COMMAND for log filename; fix for output/logs dir perms
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 4/x; better logging, check & force `DEST_LANG`
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 3/x; don't write to stderr in generated Dockerfile
  - it's `drastic red` on non-buildx dockers
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 2/x, logging
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 1/x
  - source configs in a logging section.
  - Docker: silent, fast retries to make sure `docker system df` works
  - shut-up `chown` (no `-v`) output related to  `SET_OWNER_TO_UID`
  - ask user to wait while `DESTIMG` is rsync'ed to `FINALDEST` -- it's potentially very slow
  - use green apple for Mac logging, instead of red apple which might imply error...
- WiP: the great cli entrypoint (+docker) rewrite, Phase 4: run as non-root, maybe-with-Docker
  - introduce `is_docker_ready_to_go()`; if it is, and we're not root, use Docker instead of sudo. <- GOOD IDEA? BAD IDEA? lol
  - introduce `SET_OWNER_TO_UID` var to be passed to Docker/sudo so written files are owned by the launching user, not root.
    - introduce `mkdir_recursive_and_set_uid_owner()` and `reset_uid_owner()` to reset owner based on `SET_OWNER_TO_UID`
    - use it for userpatches files created, logs, and output files, including images and debs.
  - @TODOs ref. `$SUDO_USER` which I think the old version of this?
  - add a lot of @TODOs, ref being able to relaunch something that's not `build` inside Docker, also add/change params and configs and command.
    - initially add `ARMBIAN_DOCKER_RELAUNCH_EXTRA_ARGS`
- WiP: the great cli entrypoint (+docker) rewrite, Phase 3: rpardini is demented, v3
- WiP: the great cli entrypoint (+docker) rewrite, Phase 2: rpardini is demented
- WiP: the great cli entrypoint (+docker) rewrite, Phase 1
- armbian-next: WiP: Docker: actually use the GHA-image as base; pull it every 24hs.
  - using image in my private repo.
  - this has significant speedup to "start building time" on the 1st run
  - move some Linux specific stuff to its own if
  - add comments and todo
- armbian-next: WiP: Docker, high-WiP, beginnings of Armbian mount dict, with linux/darwin preferences
- armbian-next: WiP: Docker, configure `BUILDKIT_COLORS`
- armbian-next: WiP: Docker, make docker image from Dockerfile more compact by flattening layers
- armbian-next: `logging`: add whale indicator if build running under Docker
- armbian-next: WiP: `docker`: working with `bookworm`, `sid`, and `jammy` on Darwin & Linux; works with `bullseye` on Linux only
- armbian-next: WiP: `docker`: force ARMBIAN_RUNNING_IN_CONTAINER both in Dockerfile and passed as `--env`; apt update and install in same layer; back to jammy
- armbian-next: introduce `armbian_is_running_in_container()` and `armbian_is_host_running_systemd()`, replacing `systemd-detect-virt` in multiple spots
- WiP: try with debian:bullseye -- can't detect docker at all
- armbian-next: WiP: 2nd stab at new Docker support; Darwin still works; Linux `docker.io` working
  - gen .dockerignore together with Dockerfile
  - split in funcs
  - hacks for Linux and `/dev/loop` stuff, CONTAINER_COMPAT=yes
  - mac still works, Linux stuff would break it but I if'fed
- armbian-next: the secrets of `CONTAINER_COMPAT` revealed; add size checking to check_loop_device() and avoid retry when `mknod`ing
  - this fails for the right reasons now, causing retries, which are then retried and work ;-)
  - this is related to building under Docker on Linux, using docker.io package (not docker-ce)
- armbian-next: remove `.dockerignore` and add it to `.gitignore`; it's going to be auto-generated
- armbian-next: `.dockerignore`: Docker context should only have minimal files and folders, to speed up Dockerfile build
  - IMPORTANT: `.dockerignore` is going to be generated from now on: so this is the last commit with changes before removal
-  armbian-next: WiP: initial stab at new Docker support; really run the passed cmdline; add Dockerfile to gitignore
-  armbian-next: WiP: initial stab at new Docker support; generate Dockerfile; introduce REQUIREMENTS_DEFS_ONLY
  - uses REQUIREMENTS_DEFS_ONLY
  - works on Docker Desktop on Mac;
  - linux TBA
- armbian-next: don't error out if `.git` not present; other small fixes
- armbian-next: general "work or at least don't misbehave when run on a very bare ubuntu:latest instance"
  - can't assume things, for example:
  - that `sudo` will be available; it might not, and might be already root, no reason to fail
  - that `/etc/timezone` will exist
  - that `systemd-detect-virt` will be available
  - that `git` will be available
  - that `locale-gen` will be available
2023-02-18 07:39:43 -03:00
Ricardo Pardini
2c6751f584
armbian-next: show mkfs ext2/ext4 output (remove -q), it's being run under the logging manager already 2023-02-18 07:39:42 -03:00
Ricardo Pardini
8f26db138a
armbian-next: traps: don't duplicate error message/stacktrace when error occurs in subshell; instead indicate clearly SUBSHELL 2023-02-18 07:39:41 -03:00
Ricardo Pardini
775cd6c619
armbian-next: logging: don't bomb due to lack of git or zstdmt
- logging might run super early when dependencies are not installed
- or say in a container, without git
2023-02-18 07:39:40 -03:00
Ricardo Pardini
5e55c61bfc
armbian-next: kernel: add hooks fetch_sources_for_kernel_driver and patch_kernel_for_driver to allow migrating EXTRAWIFI stuff to extensions 2023-02-18 07:39:39 -03:00
Ricardo Pardini
c4165abe4e
armbian-next: export CHOSEN_KERNEL_WITH_ARCH again, for reporting only 2023-02-18 07:39:38 -03:00
Lane Jennison
28c7645a14
armbian-next: fix old fuction name refrence cleaning to general_cleaning 2023-02-18 07:39:36 -03:00
Ricardo Pardini
cb38a8071b
armbian-next: umount_chroot_recursive: don't try to realpath unless it's a dir to begin with 2023-02-18 07:39:35 -03:00
Ricardo Pardini
4cebf28be4
armbian-next: move resolv.conf symlink to systemd to post_debootstrap_tweaks so customize phase has working DNS 2023-02-18 07:39:33 -03:00
Ricardo Pardini
3e0729b5da
armbian-next: normalize mount/umount $targets to have (or not) the trailing slash 2023-02-18 07:39:32 -03:00
Ricardo Pardini
72cf555bce
armbian-next: bye bye $FAST_CREATE_IMAGE: always use truncate for blank image 2023-02-18 07:39:31 -03:00
Ricardo Pardini
914348ba52
armbian-next: less verbose umount_chroot(), except after first try, so we can know what is left mounted 2023-02-18 07:39:30 -03:00
Ricardo Pardini
f921c8cd74
armbian-next: let all unmounting of ${MOUNT} be done by umount_chroot_recursive 2023-02-18 07:39:29 -03:00
Ricardo Pardini
d85bf2a155
armbian-next: many fixes after v29 rebase; mostly involving umount
- remove grub's --verbose, it's really too verbose
- trust in TMPDIR, do not use trap at all in compile_plymouth-theme-armbian - settings traps adhoc is forbidden in armbian-next as well
- be more verbose in umount_chroot(), do not try to unmount tmpfs-mounted dir itself, only xx/tmp
- try to be smarter about /tmp being mounted in rootfs-to-image -- in the end the recursive saves us
- run post_debootstrap_tweaks when the SDCARD is still mounted, not after
- don't try to download anything from any mirrors if SKIP_ARMBIAN_REPO==yes
2023-02-18 07:39:28 -03:00
Ricardo Pardini
833a7d9ab6
armbian-next: fix "create list of installed packages for debug" 2023-02-18 07:39:27 -03:00
Ricardo Pardini
b73e72721a
armbian-next: HACK: ROOTFSCACHE_VERSION is undefined and failing everytime 2023-02-18 07:39:26 -03:00