This commit implements a complete parallel repository management system
that allows building and publishing Debian repositories in parallel,
significantly reducing build time for multiple distributions.
- `update-main`: Builds common/main component once for all releases
- `update -R <release>`: Builds release-specific components in isolated DBs
- `merge`: Combines common + release-specific components into final repos
- Isolated databases (aptly-isolated-<release>) avoid locking during parallel builds
- Common component built once, not duplicated per release
- Release-specific components (utils, desktop) built independently
- Final merge combines all components with proper GPG signing
- Fixed GPG signing to target top-level Release files (dists/{release}/Release)
- Pool cleanup before publishing avoids "file already exists" errors
- Smart package import skips duplicates during merge
- Proper handling of empty repositories and missing components
- Improved error handling and logging throughout
1. update-main: Build common component (once)
2. update -R <release>: Parallel workers build release-specific components
3. merge: Combine all components and publish with GPG signatures
This enables GitHub Actions to run multiple release builders in parallel,
reducing total repository build time from hours to minutes.
Signed-off-by: Igor Pecovnik <igor@armbian.com>
* tools/repository/extract-repo.sh: simplify extraction by copying directly from pool
Remove dependency on Packages index files. Instead of parsing package metadata
to find file locations, directly scan the pool/ directory structure and copy all
.deb files found in each component subdirectory.
This simplifies the code and makes it more robust since it doesn't rely on
index files being present or correctly formatted.
Signed-off-by: Igor Pecovnik <igor@armbian.com>
* Add helper script: recursively clean Armbian Debian package artifacts
* Update tools/repository/cleanup-debs.sh
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
---------
Signed-off-by: Igor Pecovnik <igor@armbian.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
- Remove redundant run_cmd function, use run_aptly consistently
- Include EOS releases in repository processing (remove -ve 'eos' filter)
- Fix showall command to auto-discover repos from isolated databases
- Fix source file preservation in parallel mode (keep sources for workers)
- Fix snapshot management to handle published snapshots correctly
- Fix drop_unsupported_releases to only drop unsupported, not all repos
- Remove isolated DB cleanup that caused data loss on each run
- Remove inappropriate sudo usage from html and date commands
- Enable FORCE_PUBLISH by default for better workflow
- Improve common snapshot creation in isolated mode
- Add isolated database support to showall function
Signed-off-by: Igor Pecovnik <igor@armbian.com>
**Repository tools: enable parallel repo generation and refactor repo.sh**
Add support for parallel repository generation using GitHub Actions, allowing
multiple workers to build different releases concurrently without database
locking conflicts.
### Highlights
- Add `-R/--single-release`, `update-main`, and `merge` for CI-level parallelism
- Use isolated per-release databases; remove local `-j/--parallel-jobs`
- Sign all Release files; optimize signing order
- Add `extract-repo.sh` for extracting and reorganizing packages from repositories
- Replace file-based logging with syslog (`logger`), removing sudo requirement
- Improve error handling, signing logic, and robustness
- Refactor repo.sh for better readability and documentation
- Apply CodeRabbit-recommended fixes
Signed-off-by: Igor Pecovnik <igor@armbian.com>
* merge patches from rockchip and rk322x families for current and edge
kernels
* adjust patches for tinkerboard to remove some cruft and overclocking
* rework kernel configs
* fix rk322x dmc to avoid lockup on rk3288
* migrate rockchip-6.6 into patch series, rename all patches with
more understandable names
* add gen-series.sh script in tools directory (it is a naive tool to
create patch series when you don't want to rebase everything)
> How to use:
>
> `./compile.sh inventory` - does just the board inventory; look for output in `output/info`
>
> `./compile.sh targets-dashboard` - does inventory, targets compositing, and images info; look for output in `output/info`, read the instructions output by the command if you want to load the OpenSearch dashboards.
>
> `./compile.sh targets` - does the full targets compositing and artifacts, look for output in `output/info`
>
> If you don't have a `userpatches/targets.yaml`, _one will be provided for you_ defaulting to Jammy minimal CLI
> and Jammy xfce desktop, for all boards in all branches. You can pass filters via `TARGETS_FILTER_INCLUDE=...` to narrow.
>
- board JSON inventory:
- more generic regex parsing of variables from board files:
- all top-level (non-indented) variables are parsed and included in the JSON board inventory
- this allows us to add new variables to the board files without having to update the parser
- variables can be bare, `export` or `declare -g`, but **_must_ be quoted** (single or double) and UPPER_CASE
- some special treatment for certain variables:
- `KERNEL_TARGET` is parsed as a _comma-separated_ list of valid BRANCH'es
- `BOARD_MAINTAINER` is parsed as _space-separated_ list of valid maintainer GH usernames as `BOARD_MAINTAINERS: [...]` in the JSON
- script complains if `BOARD_MAINTAINER` is not set in core boards. Empty is still allowed.
- `HAS_VIDEO_OUTPUT="no"` causes `BOARD_HAS_VIDEO: false` in the JSON (for desktop-only inventorying, see below)
- introduce `not-eos-with-video` in `items-from-inventory` at the targets compositor
- the same as `not-eos`, but with added `BOARD_HAS_VIDEO: true` filter, see above
- introduce `TARGETS_FILTER_INCLUDE` for targets compositor
- this filters the targets _after_ compositing (but before getting image info), based on the board inventory data
- it's a comma-separated list of `key:value` pairs, which are OR-ed together
- new virtual info `BOARD_SLASH_BRANCH` post-compositing inventory for filtering of a specific BOARD/BRANCH combo (e.g. `odroidhc4/edge`)
- some interesting possible filters:
- `TARGETS_FILTER_INCLUDE="BOARD:odroidhc4"`: _only_ build a single board, all branches. JIRA [AR-1806]
- `TARGETS_FILTER_INCLUDE="BOARD_SLASH_BRANCH:odroidhc4/current"`: _only_ build a single board/branch combo
- `TARGETS_FILTER_INCLUDE="BOARD:odroidhc4,BOARD:odroidn2"`: _only_ build _two_ boards, all branches.
- `TARGETS_FILTER_INCLUDE="BOARD_MAINTAINERS:rpardini"`: build all boards and branches where rpardini is a maintainer
- `TARGETS_FILTER_INCLUDE="BOARDFAMILY:rockchip64"`: build all boards and branches in the rockchip64 family
- image-info-only variables like `LINUXFAMILY` is **not** available for filtering at this stage
- rename `config/templates` `targets-all-cli.yaml` to `targets-default.yaml`
- this is used when no `userpatches/targets.yaml` is found
- new default includes all boards vs branches for non-EOS boards
- also desktop for all boards that _don't_ have `HAS_VIDEO_OUTPUT='no``
- introduce simplified `targets-dashboard` CLI:
- does only inventory, compositing, and image info, but not artifact reducing, etc.
- ignore desktop builds in the OpenSearch indexer
- update the OpenSearch Dashboards, including new information now available
- invert the logic used for `CLEAN_INFO` and `CLEAN_MATRIX`
- defaults to `yes` now, so new users/CI don't get hit by stale caches by default
- repo pipeline CLI stuff is usually run on saved/restored artifacts for `output/info`, so don't clean by default via the CLI
* Added Debian 12 Bookworm.
* Link identical
* Make minimal truly minimal again
Move netplan.io to CLI image only
* Add netplan.io to Bullseye standard CLI
Co-authored-by: Konstantin Litvinov <klitvinov@piesoft.us>
Co-authored-by: Igor <igor@armbian.com>
* Moving repository handling from main script
- cleaning
- working
TBD: handling of parameters
* Not needed anymore
* Paramter handling, edit config in mktemp
* Typo
* Improve string handling
* Cleanining
* Compacting
* Change to backslash compatible sed replacement
* Publish all, update selected to avoid having erros at update
* Add support for html index generation
* Bugfix
* More arch
* Don't run publish if there is nothing in the incoming
* Add repo sync control file creation
* Fix hardcoded value
* sunxi-5.18: add megous patches: tag: orange-pi-5.18-20220627-1924
* tools: mk_format_patch: ignore-matching-lines git version
* sunxi-5.18: re-extracted armbian patches after being applied to 5.18.8
* sunxi-5.18: switch to version tag=v5.18.8
* sunxi-5.18: add new megous patches
* switch to v5.18.5: exclude a previously applied patch
* fix: tools/mk_format_patch: numbered=false by default
* sunxi-5.18: rebasing and extraction using the mk_format_patch script
* add a hand command line tool
* Rename folder
* Add empty readme
* Add tool for adjustin configs, set executable bit
* Add folder to labeller
Co-authored-by: Igor Pecovnik <igor.pecovnik@gmail.com>
