Commit Graph

355 Commits

Author SHA1 Message Date
Ricardo Pardini
1a7877bb2e
armbian-next: docker: warn & countdown if we can't write to cache/docker; continue by using .tmp/docker if not aborted. 2023-02-18 07:44:10 -03:00
Ricardo Pardini
8d2be2c694
armbian-next: python/pip: solve crazy with /usr/bin/pip3 by not using it (instead, python3 -m pip); don't mask errors
- "| cut" masked errors in multiple places
- include real pip3 version in python deps hash, all will be invalidated
- be less happy about hitting hash/cache, should happen often
2023-02-18 07:44:09 -03:00
Ricardo Pardini
2453722962
armbian-next: docker: pass CI env down to Docker (for GHA etc) 2023-02-18 07:44:08 -03:00
Ricardo Pardini
d4f969c67d
armbian-next: docker: don't bomb if stdin is not a terminal (eg, in CI) 2023-02-18 07:44:07 -03:00
Ricardo Pardini
d95478281b
armbian-next: fix ANSI logs, use reset at strategic places 2023-02-18 07:44:06 -03:00
Ricardo Pardini
eb30aac310
armbian-next: extensions: move 'extensions.sh' into library; replace manual source with init function
- wraps globals in `extension_manager_declare_globals()` -- and make them actually global
- whoever wrote this original code is insane, add copyright
2023-02-18 07:44:05 -03:00
Ricardo Pardini
092f9ad3b2
armbian-next: firmware-deb.sh: prefix temp dir so it's "deb-firmware" 2023-02-18 07:44:04 -03:00
Ricardo Pardini
486ba2953c
armbian-next: bsp: use prepare_temp_dir_in_workdir_and_schedule_cleanup(); bring back PACKAGES_SEARCH_ROOT_ABSOLUTE_DIRS 2023-02-18 07:44:03 -03:00
Igor
90c46e0215
armbian-next: cli: add CLI alias command rootfs-cache for creating rootfs cache 2023-02-18 07:44:02 -03:00
Ricardo Pardini
22074a0cde
armbian-next: logging: new log file / summary nomenclature 2023-02-18 07:44:01 -03:00
Ricardo Pardini
a6182bbd89
armbian-next: docker: archive logfiles again; warn & greet testers when archiving many old logfiles 2023-02-18 07:44:00 -03:00
Ricardo Pardini
dcaadac152
armbian-next: git: disallow forcing impossible shallow/full; warn if it's the case
- you gotta wipe the worktree before you change it from full to shallow or vice-versa
2023-02-18 07:43:59 -03:00
Ricardo Pardini
da77aad1dd
armbian-next: git: detect, and stop, when git bare trees vs worktrees have been shuffled around and can't be recovered
- hopefully this will never happen ;-)
2023-02-18 07:43:58 -03:00
Ricardo Pardini
51f9f59084
armbian-next: back to cleaning downloaded bundles/gitballs after they've converted to bare tree & proven working 2023-02-18 07:43:57 -03:00
Ricardo Pardini
25d5fe5926
armbian-next: darwin: try to get rid of .DS_Store sneaky bastards before launching Docker 2023-02-18 07:43:56 -03:00
Ricardo Pardini
84e0bc1423
armbian-next: fix, don't try to get pid descendants on non-Linux (fixes Darwin cleanups) 2023-02-18 07:43:55 -03:00
Ricardo Pardini
88ff66ae11
armbian-next: adaptive shallow/full Kernel git bare git (3gb -> 300mb download)
- allows using pre-shallow-ed bare trees, specific to the KERNEL_MAJOR_MINOR
- uses shallow by default if storage is "slow" (mmc), or free disk space < 32g (@TODO tweak this)
- if full tree already there keeps using it
- allow forcing decision with `KERNEL_GIT=full` or `KERNEL_GIT=shallow`
- countdown to confirm if decision based on storage type/free space and user is interactive
  - this has some of the worst copy ever written
- @TODO refactor the device type / free space code, reuse across codebase (3-4 other spots detect free space)
- @TODO GHA self-hosted (full) vs hosted (shallow) runners decision
- sprinkle with `wait_for_disk_sync()` around clean/download/extract
- sprinkle git with debugs, and an info between fetch and checkout (otherwise it seems "fetch" takes a long time)
2023-02-18 07:43:54 -03:00
Ricardo Pardini
72c61f2f68
armbian-next: replace bare sync invocations by introducing wait_for_disk_sync(); sync early at start of build
- wait 10s for sync call; otherwise warn user to be patient
- wait_for_sync "reason for sync" everywhere
- add @TODO for /run/user/0 etc and XDG_RUNTIME_DIR in chroot
2023-02-18 07:43:53 -03:00
Ricardo Pardini
4d43c05f7b
armbian-next: kernel: git: remove dead code, GIT_PRE_FETCH_HOOK / checked_out_revision_mtime 2023-02-18 07:43:52 -03:00
Ricardo Pardini
3eacae68c7
armbian-next: kernel: git: split via-ORAS and via-Bundle tangle 2023-02-18 07:43:51 -03:00
Ricardo Pardini
8af14d4f8c
armbian-next: countdown: split off file for countdown functions (countdown-to-abort and countdown-to-continue)
- introduce `countdown.sh`, move `traps.sh::exit_if_countdown_not_aborted()` there, introduce `exit_if_countdown_not_aborted()`
2023-02-18 07:43:50 -03:00
Ricardo Pardini
2c9582f9ba
armbian-next: add Debian Ports keyring hostdep to strap Debian riscv64 from ports
- rpardini: actually use ports keyring for Debian Risc-V
- `risc-v`: correctly include `debian-ports-archive-keyring` in Risc-V CLI packages
  - include in debootstrap list, via `config/optional/architectures/riscv64/_config/cli/_all_distributions/main/packages`
  - can't have two `--include` in debootstrap call
- riscv64: move `debian-ports-archive-keyring` from main to debootstrap; also only for `sid`
2023-02-18 07:43:49 -03:00
Ricardo Pardini
d8016c49b6
armbian-next: fix ATF build under binutils 2.39+ via TF_LDFLAGS=--no-warn-rwx-segment 2023-02-18 07:43:48 -03:00
Ricardo Pardini
549ae08061
armbian-next: git: show "fetch completed" info, lest user thinks fetch is hanging while checking out 2023-02-18 07:43:46 -03:00
Ricardo Pardini
47bac3af92
armbian-next: fix missing BOARD_TYPE 2023-02-18 07:43:45 -03:00
Ricardo Pardini
8cedc82ec2
armbian-next: hopefully improved error logging; avoid logging errors up to 3x; subdued red stacktraces
- reorg `display_alert()` so more frequent levels are handled first
- improve some ANSI color usage, still a mess
- don't write failed commands stackstrace as warnings to screen -- just to file (yes it's redundant, except if in test)
- more useful / subdued stacktraces, to make actual error more obvious...
2023-02-18 07:43:43 -03:00
Ricardo Pardini
da1cf2b475
armbian-next: rename too-long main function to prep_conf_main_build_single()
- this commit is proof of OCD (old name didn't fit stacktrace space)
2023-02-18 07:43:42 -03:00
Ricardo Pardini
242a5dad1f
armbian-next: somewhat-better handling/logging when using Docker (don't show two logfiles to inspect)
- do not show path to Docker run-host's logfile, unless the Docker run itself didn't produce logs
- introduce `global_final_exit_code` to allow to maintain/preserve Docker's exit code without triggering an error host-side
2023-02-18 07:43:41 -03:00
Ricardo Pardini
bf3f346bee
armbian-next: still fighting tee leaking: now I think I won, again
- avoid using the "last resort, use lazy umount" by killing tee_pid _and descendants_ before hand. Works!
- introduce `get_descendants_of_pid_array()` using recursive trick to get all descendants
- do not `unset CURRENT_LOGFILE` -- during `logging_error_show_log()` (from SHOW_LOG=no days)
  - do `unset CURRENT_LOGFILE` immediately before processing logs -- although it magically survives. mysteries of traps in bash...
- `declare -g` instead of `export` for `start_logging_section()`
- don't `check_and_close_fd_13()` in `run_cleanup_handlers()`
- do run `check_and_close_fd_13()`, but only after archiving the old logs (in logging trap!)
- add a default error for unhandled TRAP's -- of which should be done
- fix reset of `global_tee_pid=0` in section-logging
2023-02-18 07:43:40 -03:00
Ricardo Pardini
c1100fa461
armbian-next: still fighting tee leaking under duress, turns out I _hadn't_ really won
- last resort, use lazy umount.
- include `lsof` in hostdeps, useful to debug these situations
2023-02-18 07:43:39 -03:00
Ricardo Pardini
a3d8b5891b
armbian-next: aggregation.py vs create-cache.sh: better hashing, much desktop
- aggregation.py:
  - only include non-removed pkgs in hash (do'h)
  - use a sorted unique of desktop+rootfs+debootstrap pkgs for pkg hash
  - make space for having apt sources/gpg keys in hash @TODO
  - write a hash-text variable for bash with info, for easy comparison later
  - refactor out bash_string_multiline()
- create-cache.sh:
  - don't armhf unless `ARMHF_ARCH=yes`
  - remove the uninstall stuff until we decide @TODO
  - don't show "free space of MOUNT" when there's only SDCARD
  - better logging for resolv.conf/machine-id shenanigans does in rootfs cache
  - drop dead code, squash some quoting issues
  - show the size of the zstd ball after compressing
  - don't even go to github armbian/cache nor cache.armbian.com for list of caches if SKIP_ARMBIAN_REPO=yes
  - remove mentions of BOARD or BRANCH, not relevant in rootfs cache
2023-02-18 07:43:36 -03:00
Ricardo Pardini
be49433b41
armbian-next: use retries for downloading ORAS tooling
- `do_with_retries()`: add `IS_A_RETRY` and `RETRY_FMT_MORE_THAN_ONCE` for convenient logging in retried functions
2023-02-18 07:43:35 -03:00
Ricardo Pardini
ec7639e518
armbian-next: severe bug, there was no prefix-emoji for Windows 2023-02-18 07:43:33 -03:00
Ricardo Pardini
4dd278f07d
armbian-next: compile_firmware(): use new temp dir helpers (saves 2Gb+ in WORKDIR), fix typos 2023-02-18 07:43:32 -03:00
Ricardo Pardini
62d948b1b4
armbian-next: still fighting tee leaking under duress, and I think I won
- call `check_and_close_fd_13()` directly when `exit_with_error()` called, don't wait for trap
- store PID spawned by `do_with_logging()` under `$global_tee_pid`
- `check_and_close_fd_13()`:
  - sprinkle with `sync` to give a chance for stuff to die gracefully
  - check if `$global_tee_pid` is set and actually a running PID, if so, kill it
  - do not blindly "wait" for stuff, instead, kill it and wait for it to die -- 100% more likely to not hang ;-)
2023-02-18 07:43:31 -03:00
Ricardo Pardini
ce47db1e0a
armbian-next: add logging with size of actual built rootfs in MiB; debug tmpfs in trap & after pkgs done 2023-02-18 07:43:30 -03:00
Ricardo Pardini
164db02af5
armbian-next: small cleanups: squash typos / add function keyword / add types / mark dead code 2023-02-18 07:43:29 -03:00
Ricardo Pardini
35fbb7ed6c
armbian-next: small cleanups: squash typos / add function keyword / add types / mark dead code 2023-02-18 07:43:28 -03:00
Ricardo Pardini
0ea0fe3528
armbian-next: mktemp and tmpfs related utility functions with automatic cleanup handlers 2023-02-18 07:43:27 -03:00
Ricardo Pardini
af7ace7e57
armbian-next: don't leak .raw image if failure before move; show rootfs-tmpfs info
- add escapings and TODO's around the area...
2023-02-18 07:43:26 -03:00
Ricardo Pardini
1636a2afea
armbian-next: tune logging: unmounting, u-boot prep/source/install; traps; extensions 2023-02-18 07:43:25 -03:00
Ricardo Pardini
ff4b5069ce
armbian-next: extract move_images_to_final_destination() and optimize for fast-move when src/dst on same filesystem 2023-02-18 07:43:24 -03:00
Ricardo Pardini
92d902c883
armbian-next: drop dead/duplicated code for bootsplash and mark kernel-drivers.sh (which includes bootsplash) as dead code 2023-02-18 07:43:23 -03:00
Ricardo Pardini
4d59ad7ac4
armbian-next: try harder to get en_US.UTF-8 logging 2023-02-18 07:43:22 -03:00
Ricardo Pardini
99a7a7980b
armbian-next: docker: remove old code 2023-02-18 07:43:21 -03:00
Ricardo Pardini
a6a5152695
armbian-next: chroot_sdcard_apt_get_update() to replace and better log chroot_sdcard_apt_get update
- also add retries where it was missing
- @TODO: desktop_postinstall() is a mess, needs aggregation work
2023-02-18 07:43:19 -03:00
Ricardo Pardini
2455c55dac
armbian-next: firmware: don't build -full firmware if not on CI/noninteractive and board's not going to use it
- move `fakeroot_dpkg_deb_build()` from runners.sh to new utils-dpkgdeb.sh
  - and make it debug show the size of the source
- clean off `.git` directory earlier for full build
- @TODO: get rid of the "hardlinks" funky biz in there and not-copy `.git` to begin with
2023-02-18 07:43:18 -03:00
Ricardo Pardini
f2bd68afde
armbian-next: kernel: cleanup bundle after patching succeeded, not after build success
- for extra gigabytes of space, earlier
2023-02-18 07:43:17 -03:00
Ricardo Pardini
48c0e9acf4
armbian-next: severe bug, countdown counted up 2023-02-18 07:43:16 -03:00
Ricardo Pardini
8444e7e308
armbian-next: bring back swig hostdep -- needed for some u-boot's 2023-02-18 07:43:15 -03:00