Commit Graph

10894 Commits

Author SHA1 Message Date
Ricardo Pardini
d24d3327a8
armbian-next: the great cli entrypoint (+docker) rewrite; introduce USE_LOCAL_APT_DEB_CACHE replacing apt-cacher-ng
- armbian-next: introduce `USE_LOCAL_APT_DEB_CACHE` (default `=yes`) as alternative/in addition to `apt-cacher-ng` (eg, in Docker)
  - this uses `cache/aptcache/${RELEASE}-${ARCH}` (in the host) for
      - apt cache, by bind-mounting it to `${SDCARD}/var/cache/apt` in the `chroot_sdcard_apt_get()` runner and its usages
      - debootstrap, by passing it `--cache-dir`
  - utility function to help understand what is happening to cache during usage
  - apt itself mantains this cache, removing old packages when new ones are installed. apt does this _by default_
      - introduce `DONT_MAINTAIN_APT_CACHE=yes` to skip out of automatic apt maintenance of apt cache, eg, during `remove`s
      - don't do `apt clean` and such if using local cache, that would clean the cache, not the chroot
  - clean up `install_deb_chroot()` a little, find an unrelated bug there
- WiP: the great cli entrypoint (+docker) rewrite, Phase 6: relaunching structure; re-pass ARMBIAN_BUILD_UUID; use ARMBIAN_COMMAND for log filename; fix for output/logs dir perms
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 4/x; better logging, check & force `DEST_LANG`
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 3/x; don't write to stderr in generated Dockerfile
  - it's `drastic red` on non-buildx dockers
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 2/x, logging
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 1/x
  - source configs in a logging section.
  - Docker: silent, fast retries to make sure `docker system df` works
  - shut-up `chown` (no `-v`) output related to  `SET_OWNER_TO_UID`
  - ask user to wait while `DESTIMG` is rsync'ed to `FINALDEST` -- it's potentially very slow
  - use green apple for Mac logging, instead of red apple which might imply error...
- WiP: the great cli entrypoint (+docker) rewrite, Phase 4: run as non-root, maybe-with-Docker
  - introduce `is_docker_ready_to_go()`; if it is, and we're not root, use Docker instead of sudo. <- GOOD IDEA? BAD IDEA? lol
  - introduce `SET_OWNER_TO_UID` var to be passed to Docker/sudo so written files are owned by the launching user, not root.
    - introduce `mkdir_recursive_and_set_uid_owner()` and `reset_uid_owner()` to reset owner based on `SET_OWNER_TO_UID`
    - use it for userpatches files created, logs, and output files, including images and debs.
  - @TODOs ref. `$SUDO_USER` which I think the old version of this?
  - add a lot of @TODOs, ref being able to relaunch something that's not `build` inside Docker, also add/change params and configs and command.
    - initially add `ARMBIAN_DOCKER_RELAUNCH_EXTRA_ARGS`
- WiP: the great cli entrypoint (+docker) rewrite, Phase 3: rpardini is demented, v3
- WiP: the great cli entrypoint (+docker) rewrite, Phase 2: rpardini is demented
- WiP: the great cli entrypoint (+docker) rewrite, Phase 1
- armbian-next: WiP: Docker: actually use the GHA-image as base; pull it every 24hs.
  - using image in my private repo.
  - this has significant speedup to "start building time" on the 1st run
  - move some Linux specific stuff to its own if
  - add comments and todo
- armbian-next: WiP: Docker, high-WiP, beginnings of Armbian mount dict, with linux/darwin preferences
- armbian-next: WiP: Docker, configure `BUILDKIT_COLORS`
- armbian-next: WiP: Docker, make docker image from Dockerfile more compact by flattening layers
- armbian-next: `logging`: add whale indicator if build running under Docker
- armbian-next: WiP: `docker`: working with `bookworm`, `sid`, and `jammy` on Darwin & Linux; works with `bullseye` on Linux only
- armbian-next: WiP: `docker`: force ARMBIAN_RUNNING_IN_CONTAINER both in Dockerfile and passed as `--env`; apt update and install in same layer; back to jammy
- armbian-next: introduce `armbian_is_running_in_container()` and `armbian_is_host_running_systemd()`, replacing `systemd-detect-virt` in multiple spots
- WiP: try with debian:bullseye -- can't detect docker at all
- armbian-next: WiP: 2nd stab at new Docker support; Darwin still works; Linux `docker.io` working
  - gen .dockerignore together with Dockerfile
  - split in funcs
  - hacks for Linux and `/dev/loop` stuff, CONTAINER_COMPAT=yes
  - mac still works, Linux stuff would break it but I if'fed
- armbian-next: the secrets of `CONTAINER_COMPAT` revealed; add size checking to check_loop_device() and avoid retry when `mknod`ing
  - this fails for the right reasons now, causing retries, which are then retried and work ;-)
  - this is related to building under Docker on Linux, using docker.io package (not docker-ce)
- armbian-next: remove `.dockerignore` and add it to `.gitignore`; it's going to be auto-generated
- armbian-next: `.dockerignore`: Docker context should only have minimal files and folders, to speed up Dockerfile build
  - IMPORTANT: `.dockerignore` is going to be generated from now on: so this is the last commit with changes before removal
-  armbian-next: WiP: initial stab at new Docker support; really run the passed cmdline; add Dockerfile to gitignore
-  armbian-next: WiP: initial stab at new Docker support; generate Dockerfile; introduce REQUIREMENTS_DEFS_ONLY
  - uses REQUIREMENTS_DEFS_ONLY
  - works on Docker Desktop on Mac;
  - linux TBA
- armbian-next: don't error out if `.git` not present; other small fixes
- armbian-next: general "work or at least don't misbehave when run on a very bare ubuntu:latest instance"
  - can't assume things, for example:
  - that `sudo` will be available; it might not, and might be already root, no reason to fail
  - that `/etc/timezone` will exist
  - that `systemd-detect-virt` will be available
  - that `git` will be available
  - that `locale-gen` will be available
2023-02-18 07:39:43 -03:00
Ricardo Pardini
2c6751f584
armbian-next: show mkfs ext2/ext4 output (remove -q), it's being run under the logging manager already 2023-02-18 07:39:42 -03:00
Ricardo Pardini
8f26db138a
armbian-next: traps: don't duplicate error message/stacktrace when error occurs in subshell; instead indicate clearly SUBSHELL 2023-02-18 07:39:41 -03:00
Ricardo Pardini
775cd6c619
armbian-next: logging: don't bomb due to lack of git or zstdmt
- logging might run super early when dependencies are not installed
- or say in a container, without git
2023-02-18 07:39:40 -03:00
Ricardo Pardini
5e55c61bfc
armbian-next: kernel: add hooks fetch_sources_for_kernel_driver and patch_kernel_for_driver to allow migrating EXTRAWIFI stuff to extensions 2023-02-18 07:39:39 -03:00
Ricardo Pardini
c4165abe4e
armbian-next: export CHOSEN_KERNEL_WITH_ARCH again, for reporting only 2023-02-18 07:39:38 -03:00
Ricardo Pardini
40e3b642d6
armbian-next: don't break if parsing BOOT_SOC out of BOOTCONFIG fails; sanity checks for BOOT_SOC/DDR_BLOB
- (eg: `none` case: `jetson-nano`)
- but add sanity checks for BOOT_SOC/DDR_BLOB etc
- and fail if BOOT_SOC is not defined when function runs
2023-02-18 07:39:37 -03:00
Lane Jennison
28c7645a14
armbian-next: fix old fuction name refrence cleaning to general_cleaning 2023-02-18 07:39:36 -03:00
Ricardo Pardini
cb38a8071b
armbian-next: umount_chroot_recursive: don't try to realpath unless it's a dir to begin with 2023-02-18 07:39:35 -03:00
Ricardo Pardini
c444040740
armbian-next: grub: call VER="generic" update_initramfs when "${DISTRO_GENERIC_KERNEL}" == "yes" (so ddk builds work again) 2023-02-18 07:39:34 -03:00
Ricardo Pardini
4cebf28be4
armbian-next: move resolv.conf symlink to systemd to post_debootstrap_tweaks so customize phase has working DNS 2023-02-18 07:39:33 -03:00
Ricardo Pardini
3e0729b5da
armbian-next: normalize mount/umount $targets to have (or not) the trailing slash 2023-02-18 07:39:32 -03:00
Ricardo Pardini
72cf555bce
armbian-next: bye bye $FAST_CREATE_IMAGE: always use truncate for blank image 2023-02-18 07:39:31 -03:00
Ricardo Pardini
914348ba52
armbian-next: less verbose umount_chroot(), except after first try, so we can know what is left mounted 2023-02-18 07:39:30 -03:00
Ricardo Pardini
f921c8cd74
armbian-next: let all unmounting of ${MOUNT} be done by umount_chroot_recursive 2023-02-18 07:39:29 -03:00
Ricardo Pardini
d85bf2a155
armbian-next: many fixes after v29 rebase; mostly involving umount
- remove grub's --verbose, it's really too verbose
- trust in TMPDIR, do not use trap at all in compile_plymouth-theme-armbian - settings traps adhoc is forbidden in armbian-next as well
- be more verbose in umount_chroot(), do not try to unmount tmpfs-mounted dir itself, only xx/tmp
- try to be smarter about /tmp being mounted in rootfs-to-image -- in the end the recursive saves us
- run post_debootstrap_tweaks when the SDCARD is still mounted, not after
- don't try to download anything from any mirrors if SKIP_ARMBIAN_REPO==yes
2023-02-18 07:39:28 -03:00
Ricardo Pardini
833a7d9ab6
armbian-next: fix "create list of installed packages for debug" 2023-02-18 07:39:27 -03:00
Ricardo Pardini
b73e72721a
armbian-next: HACK: ROOTFSCACHE_VERSION is undefined and failing everytime 2023-02-18 07:39:26 -03:00
Ricardo Pardini
9137a9de3f
armbian-next: deploy bootscript even if BOOTCONFIG=none
- fact we're not BUILDING nor deploying uboot does not mean image does not need a bootscript.
2023-02-18 07:39:25 -03:00
Ricardo Pardini
f052196b6c
armbian-next: debugs about armbianEnv.txt overlays/fdtfile 2023-02-18 07:39:24 -03:00
Ricardo Pardini
7b47e79780
armbian-next: introduce ARMBIAN_LOGS_TO_JOURNAL and ARMBIAN_LOGS_JOURNAL_IDENTIFIER (logs display_alert to journald) 2023-02-18 07:39:23 -03:00
Ricardo Pardini
19938837b5
armbian-next: re-add dependency fdisk after juggling rebases 2023-02-18 07:39:22 -03:00
Ricardo Pardini
38e2e50328
armbian-next: split early_prepare_host_dependencies() (run early, during config, before aggregation) from prepare_host() (run after aggregation)
- add hook `host_dependencies_known` (different from `host_dependencies_ready` from before)
2023-02-18 07:39:21 -03:00
Ricardo Pardini
1fa825b58e
armbian-next: fix for tests of BOOTSCRIPT and BOOTCONFIG!=none 2023-02-18 07:39:20 -03:00
Ricardo Pardini
271fe3fcd0
armbian-next: retry apt update & apt install family packages in addition to board packages 2023-02-18 07:39:19 -03:00
Ricardo Pardini
45c2728851
armbian-next: prepare a bin dir inside WORKDIR with a python symlink to /usr/bin/python2 and add it to PATH 2023-02-18 07:39:18 -03:00
Ricardo Pardini
4055399d15
armbian-next: sources/families: meson64: gx: use python2 explicitly for calling acs_tool.pyc; better logging 2023-02-18 07:39:17 -03:00
Ricardo Pardini
a1ff891b24
armbian-next: cleaning: don't complain about 'none' CLEAN_LEVEL 2023-02-18 07:39:16 -03:00
Ricardo Pardini
188ef0b1af
armbian-next: kernel: run kernel's make through unbuffer as well 2023-02-18 07:39:15 -03:00
Ricardo Pardini
98eae22665
armbian-next: uboot: use CFLAGS/KCFLAGS everywhere; run make through unbuffer (from expect package) for full color logging 2023-02-18 07:39:14 -03:00
Ricardo Pardini
2196e957f0
armbian-next: logs: avoid errors by cwd'ing to SRC 2023-02-18 07:39:13 -03:00
Ricardo Pardini
379b7d6f19
armbian-next: write_image_to_device: if SKIP_VERIFY=no, don't waste time checksumming the image, it won't be used 2023-02-18 07:39:12 -03:00
Ricardo Pardini
a02647862a
armbian-next: u-boot: add debug info when hacking CONFIG_ENV_IS_IN_EXT4 in u-boot .config 2023-02-18 07:39:11 -03:00
Ricardo Pardini
bfa5813c6e
armbian-next: u-boot: add fasthash to u-boot [deprecated, later removed]
- u-boot: fasthash: add `uboot_target_counter` to id and branch, lest `UBOOT_TARGET_MAP` > 2 fails
2023-02-18 07:39:10 -03:00
Ricardo Pardini
f07ec572d8
armbian-next: fix logging headers in HTML/ANSI log output 2023-02-18 07:39:09 -03:00
Ricardo Pardini
578020f42c
armbian-next: cleaning: fix shortcircuits, warn about unknown clean levels 2023-02-18 07:39:08 -03:00
Ricardo Pardini
b0cb5c270f
armbian-next: traps: cleanups: run cleanups in the reverse order they were added; allow running one single cleanup and removing from list 2023-02-18 07:39:07 -03:00
Ricardo Pardini
d6c43d0cdf
armbian-next: kernel-localmodconfig: update kernel_config_mtime after running localmodconfig, otherwise changes not detected 2023-02-18 07:39:06 -03:00
Ricardo Pardini
41e471aa1e
armbian-next: introduce KERNEL_EXTRA_DIR to suffix LINUXSOURCEDIR forcefully 2023-02-18 07:39:05 -03:00
Ricardo Pardini
7643c2901d
armbian-next: check_loop_device() now uses do_with_retries 5 ... to give the OS time
- prevents spurious fails I get sometimes...
2023-02-18 07:39:04 -03:00
Ricardo Pardini
2ec5619745
armbian-next: python: info: replace instances of ${SRC} with literal ${SRC} so infos are more comparable 2023-02-18 07:39:03 -03:00
Ricardo Pardini
3146421f8e
armbian-next: u-boot: extract into functions; add hooks - for more flexibility
- extract loop_over_uboot_targets_and_do(), and handle `target_make/target_files` parsing there, so `compile_uboot_target` needs no parsing magic
- extract deploy_built_uboot_bins_for_one_target_to_packaging_area() (meant to be called via loop_over_uboot_targets_and_do)
- add hook `post_uboot_custom_postprocess` -- literally run after `uboot_custom_postprocess`
- @TODO: possibly squash with a previous u-boot hook was added
2023-02-18 07:39:02 -03:00
Ricardo Pardini
8ce52db74e
armbian-next: back to ANSI logs by default; HTML logs only by EXPORT_HTML_LOG=yes 2023-02-18 07:39:01 -03:00
Ricardo Pardini
c7713ebb70
armbian-next: atf: add LOG_LEVEL=40 (up from default 20) and BUILD_STRING="armbian" to atf make invocation 2023-02-18 07:39:00 -03:00
Ricardo Pardini
8d912e59c4
armbian-next: u-boot: introduce UBOOT_DEBUGGING=yes which enables CONFIG_LOG=y and others
- also pass CFLAGS as well as KCFLAGS to make
2023-02-18 07:38:59 -03:00
Ricardo Pardini
3c5021b438
armbian-next: allow overriding rkbins repo URL with RKBIN_GIT_URL 2023-02-18 07:38:58 -03:00
Ricardo Pardini
6d2c00613c
armbian-next: try to parse Source: linux-5.x.y-rcZ correctly into VER=5.x.y-rcZ instead of VER=5.x.y
- `VER` gets used in the final *image* name produced, and it is very confusing to get `5.19.0` instead of `5.19.0-rc3` there.
- added debug's so we can find when we're bit, there's 2 variations
2023-02-18 07:38:57 -03:00
Ricardo Pardini
95e0b9d484
armbian-next: remove warm/cold bundles from fetch_from_repo, replace with 2 invocation-specific hooks and gitballs (.tar of .git)
- drop experiment 'caches/gitbundles' and its code; all lives now in separate repo
- change LINUXSOURCEDIR **again**, cleanup the old one
2023-02-18 07:38:56 -03:00
Ricardo Pardini
0aa57b5ef5
armbian-next: do not exclude 'CC' messages from kernel make 2023-02-18 07:38:55 -03:00
Ricardo Pardini
8511061d67
armbian-next: kernel build: always add container (series/patch-dir) mtime as minimum mtime for patch results
- this avoids snafu's when removing patches from the directory: the file won't exist, but the parent mtime will have been updated
2023-02-18 07:38:54 -03:00