diff options
| author | Danilo M. <danix@danix.xyz> | 2026-07-14 16:59:44 +0200 |
|---|---|---|
| committer | Danilo M. <danix@danix.xyz> | 2026-07-14 16:59:44 +0200 |
| commit | 3cd610c212076d438f2d1103e0fc584a310011d5 (patch) | |
| tree | b82bd71ff139408dc84f685345862f944653966b | |
| download | sbo-dockerbuild-3cd610c212076d438f2d1103e0fc584a310011d5.tar.gz sbo-dockerbuild-3cd610c212076d438f2d1103e0fc584a310011d5.zip | |
initial commit: docker test-build toolchain
Extracted from the .extras/ dir of the sbo-slackbuilds package repo, where it
grew from a throwaway helper into a standalone tool. Git history starts fresh;
the original 41-commit development log is preserved in HISTORY.md.
Contents: the test-build CLI (resolves a local SlackBuild's deps from a
configured SBo tree and builds it in a throwaway container, lints, caches deps),
the image-builder chain that produces the images it consumes, their pure-logic
self-checks, design specs/plans, an install.sh for ~/bin, and docs.
Licensed GPLv2-only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
| -rw-r--r-- | .gitignore | 1 | ||||
| -rw-r--r-- | CLAUDE.md | 153 | ||||
| -rw-r--r-- | HISTORY.md | 54 | ||||
| -rw-r--r-- | LICENSE | 338 | ||||
| -rw-r--r-- | README.md | 85 | ||||
| -rw-r--r-- | docs/plans/2026-07-13-docker-test-build.md | 1579 | ||||
| -rw-r--r-- | docs/plans/2026-07-13-image-builder.md | 790 | ||||
| -rw-r--r-- | docs/specs/2026-07-13-docker-test-build-design.md | 258 | ||||
| -rw-r--r-- | docs/specs/2026-07-13-image-builder-design.md | 216 | ||||
| -rw-r--r-- | image-builder/README | 69 | ||||
| -rwxr-xr-x | image-builder/bootstrap.sh | 564 | ||||
| -rwxr-xr-x | image-builder/build-full-image.sh | 245 | ||||
| -rwxr-xr-x | image-builder/build-sbo-testbuild.sh | 143 | ||||
| -rw-r--r-- | image-builder/config | 15 | ||||
| -rw-r--r-- | image-builder/lib.sh | 84 | ||||
| -rwxr-xr-x | image-builder/test-image-builder.sh | 80 | ||||
| -rwxr-xr-x | install.sh | 41 | ||||
| -rw-r--r-- | overrides.example | 16 | ||||
| -rwxr-xr-x | test-build | 876 | ||||
| -rw-r--r-- | test-build-config.example | 18 | ||||
| -rw-r--r-- | test-logic.sh | 227 |
21 files changed, 5852 insertions, 0 deletions
diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..72b17c9 --- /dev/null +++ b/.gitignore @@ -0,0 +1 @@ +HANDOFF.md diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..9d4cd5f --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,153 @@ +# sbo-dockerbuild + +Docker-based toolchain for test-building [SlackBuilds.org (SBo)](https://slackbuilds.org) +packages on Slackware, on both `-current` and `15.0`, in throwaway containers. + +Extracted from the `.extras/` dir of the **sbo-slackbuilds** package repo (danix's +SBo maintenance mirror) once it grew from a helper into a standalone tool. Git +history starts fresh here; the original 41-commit log is preserved in +[HISTORY.md](HISTORY.md) (those hashes refer to sbo-slackbuilds, not this repo). + +## What this repo is (and is not) + +This is the **tooling**, not packages. It builds and consumes docker images to +verify that a SlackBuild still builds; it does not itself contain SlackBuilds. +The packages live in the maintenance repo (sbo-slackbuilds) and other repos. + +Two cooperating tools, one shared set of images: + +- **`test-build`** — verify a single SlackBuild builds cleanly in a container. +- **`image-builder/`** — build the images `test-build` consumes. + +## Layout + +``` +test-build the test-build tool (single bash script) +install.sh copy test-build into ~/bin (BINDIR= to override) +test-logic.sh pure-logic self-check for test-build (no docker) +test-build-config.example copy to ~/.config/sbo-testbuild/config and edit +overrides.example current-vs-stable dep overrides (optional) +image-builder/ bootstrap.sh, build-full-image.sh, + build-sbo-testbuild.sh, lib.sh, config, + test-image-builder.sh, README +docs/specs, docs/plans design specs and implementation plans +``` + +Everything is at the repo root (flat), not under `.extras/` — that nesting was a +sbo-slackbuilds convention and does not apply here. + +## test-build + +Resolves and builds a SlackBuild in a disposable Slackware container, then lints +it. The **target** is the local SlackBuild under edit; its **dependencies** come +from a configured SBo tree. These are two different sources — do not conflate +them (an earlier bug resolved the target from the tree, testing the published +version instead of the edit). + +### Target resolution (`resolve_target_dir`) + +Repo-agnostic, driven by where you are: + +- a **path** (absolute, `./x`, or containing `/`): that directory +- a **bare name**: `./<name>/` under CWD, else CWD itself if it *is* `<name>/` + +So `cd pkgdir && test-build <name>`, `test-build /path/to/pkg`, and +`test-build ./pkg` all work. + +### Dependency resolution + +Deps are resolved from the SBo tree in the config (`SBO_TREE_CURRENT` / +`SBO_TREE_STABLE`), topologically sorted, cycle-checked. A dep is satisfied by: + +1. building it from the tree (recursively), or +2. `installed_in_base` — already present in the full-install image (queried once + per run from the image's `/var/log/packages`), or +3. an override (see below). + +Deps build first and are cached; the target always rebuilds (that is the test). + +### current-vs-stable overrides + +`~/.config/sbo-testbuild/overrides` (see `overrides.example`), applied **only on +-current** (15.0 is the SBo baseline). Verbs: + +- `drop: <dep>` — a 15.0-only SBo dep whose payload the -current base already + ships (e.g. `rust-opt`, `google-go-lang`, `cmake-opt`, `llvm-opt`, + `python3-installer`, `python3-build`). Satisfied at resolution: not built, not + UNMET. +- `rename: <old> -> <new>` — dep renamed on -current. +- `fetch: <prog>` — dep removed from the -current tree; built via `sbopkg` in + the container. + +### The build container + +One `docker run --rm -i` per package (the `-i` is required — without it `bash -s` +gets no stdin and the heredoc is silently discarded). Inside, as root: +installs already-built deps, sources `/etc/profile.d/*.sh` (so dep-provided env +like google-go-lang's `GOROOT`/`PATH` is live in the non-login shell), builds the +SlackBuild, `installpkg`s it, and runs `sbopkglint` **in-container** (it shells +out to sudo, which only works as root here; never lint on the host). The host +reads a `LINT-CLEAN`/`LINT-FINDINGS` marker from the log. + +### Dep cache + +Host dir (`PKG_CACHE`), namespaced `<variant>-<image-digest>/`. An image rebuild +changes the digest and self-invalidates the cache. Each run prunes superseded +digests **of the same variant only**, so a `--current` run does not wipe the +`15.0` cache (both live caches survive; testing both trees back-to-back reuses +built deps). Only **deps** are cached, never the target. + +### Flags + +`--stable` (15.0), `--dry-run`, `--yes`, `--no-cache`, `--keep` (copy the built +target `.txz` to `<data>/kept/` so it can be installed on the host, e.g. to +regenerate post-install artifacts), `--no-color`. + +## image-builder + +Three chained scripts build the images, each self-gated so an unchanged night is +a cheap no-op: + +- `bootstrap.sh` → `sbo-base:{ver}` FROM scratch (gated on the mirror's + `ChangeLog.txt` hash) +- `build-full-image.sh` → `sbo-full:{ver}` all series (gated on base digest; + `--force` prunes the build cache first) +- `build-sbo-testbuild.sh` → `sbo-testbuild:{ver}` + sbopkg + maintainer-tools + (gated on full digest + tools `.txz` hash) + +All settings in `image-builder/config`. Full setup (NFS mirror mounts, LAN +registry, prebuilt pkgs, cron) is in `image-builder/README`. + +**Main-loop gotcha:** each script runs its `build_variant` in an `if ! (...)` +condition, which suppresses `set -e` inside the subshell. So `docker build` / +`docker push` failures are caught with **explicit** `if ! ...; return 1` checks, +not by relying on `set -e`. Keep that pattern when editing. + +## Testing + +Pure-logic self-checks, no docker, no network: + +```bash +bash test-logic.sh # test-build resolver/overrides/cache +bash image-builder/test-image-builder.sh # image-builder lib.sh logic +``` + +Non-trivial logic changes get a check here. The docker build path itself is +verified by running `test-build` against a real image. + +## Conventions + +- Plain bash, single-file scripts where practical, minimal dependencies. +- `set -euo pipefail` in executables; guard `set -e` around subshell conditions + (see the image-builder gotcha). +- GPLv2-only. Every source file carries a short GPL header notice. +- No em dashes in prose. + +## Deployment + +The image-builder scripts run on the docker host VM (`docker.noland.dnx`, +reached via the `docker` SSH host alias, root login), deployed under +`/opt/sbo-testbuild/image-builder/`. A nightly cron rebuilds both variants after +the repo syncs. Treat the VM as outward-facing: confirm before deploying, and +inspect a target before overwriting it. After editing a script here, sync it to +the VM so the cron runs the current version. diff --git a/HISTORY.md b/HISTORY.md new file mode 100644 index 0000000..8e12890 --- /dev/null +++ b/HISTORY.md @@ -0,0 +1,54 @@ +# History + +This project began as a side-project inside the **sbo-slackbuilds** repository +(danix's SlackBuilds.org maintenance mirror), under `.extras/`. It grew from a +throwaway docker test-build helper into a structured, self-maintaining toolchain, +so it was extracted here to be developed on its own. + +Git history starts fresh in this repo. The 41 commits below are the original +development history from sbo-slackbuilds, preserved here as a record. The hashes +refer to the sbo-slackbuilds repository, not this one. + +## Original commits (from sbo-slackbuilds, oldest first) + +- `b4c286e` 2026-07-13 — docs: test-build design spec; refine CLAUDE.md purpose and docker workflow +- `1e42a7f` 2026-07-13 — docs: test-build implementation plan +- `eb74944` 2026-07-13 — test-build: scaffold script + self-check +- `1a39e22` 2026-07-13 — test-build: config, arg parsing, version selection +- `b828163` 2026-07-13 — test-build: port SBo lookup + dependency resolver +- `3680234` 2026-07-13 — test-build: current-vs-stable override engine +- `5d1fb3a` 2026-07-13 — test-build: port dep cache, namespaced by image digest +- `f4dff4e` 2026-07-13 — test-build: port BLOCKED-BY-DEP propagation +- `44f0338` 2026-07-13 — test-build: container build step (build_one via docker) +- `4a3d6a6` 2026-07-13 — test-build: run_target orchestration + summary + confirm +- `7c7e549` 2026-07-13 — test-build: wire main, image-presence check, target lookup +- `11153a8` 2026-07-13 — test-build: fix empty-order corruption when all deps dropped +- `71b58e1` 2026-07-13 — test-build: document installed_in_base deferral +- `d9cd1ad` 2026-07-13 — image-builder: add design spec +- `4db7f46` 2026-07-13 — image-builder: add implementation plan +- `b844135` 2026-07-13 — image-builder: config + lib.sh helpers +- `8b4c556` 2026-07-13 — image-builder: curl -sS in fetch for http error diagnostics +- `3e79c1b` 2026-07-13 — image-builder: lib.sh self-check +- `b4f19b7` 2026-07-13 — image-builder: bootstrap.sh base image from NAS +- `b954189` 2026-07-13 — image-builder: bootstrap.sh cleanup trap on EXIT not RETURN +- `88f399e` 2026-07-13 — image-builder: build-full-image.sh +- `160ab1f` 2026-07-13 — image-builder: build-sbo-testbuild.sh +- `0b42333` 2026-07-13 — image-builder: guard FULL_DIGEST recompute against set -e abort +- `26674aa` 2026-07-13 — image-builder: per-variant isolation, non-zero exit on failure +- `bb67cd4` 2026-07-13 — image-builder: README with VM setup checklist +- `7363f45` 2026-07-13 — image-builder: mark scripts executable +- `e513a3e` 2026-07-13 — image-builder: require_mount in changelog gate so unmounted NAS fails loud +- `f273077` 2026-07-13 — image-builder: accept any Slackware pkg extension (*.t?z) not just .txz +- `416ea5d` 2026-07-13 — image-builder: fix false GPG-update warning from SIGPIPE on yes pipe +- `e887d1e` 2026-07-13 — image-builder: feed full build from LAN NFS mirror over HTTP, fix GPG verify +- `5e93772` 2026-07-14 — test-build: implement installed_in_base from image package db +- `d839dd4` 2026-07-14 — test-build: make drop: satisfy a dep at resolution time +- `be7880b` 2026-07-14 — test-build: fix container stdin and move lint in-container +- `3a99bb5` 2026-07-14 — test-build: auto-pull the image from the LAN registry when missing +- `7df4fdf` 2026-07-14 — image-builder: README cron matches deployed 06:00/06:20/08:00 schedule +- `78db185` 2026-07-14 — image-builder: cron gates both variants, add 15.0, drop --force +- `e4edbf2` 2026-07-14 — image-builder: cron moved to 03:00/05:00 (trees now sync 01:00/02:00) +- `e129f08` 2026-07-14 — test-build: target the local SlackBuild (CWD/path), deps from the tree +- `2b75214` 2026-07-14 — image-builder: catch build/push failures, prune cache on --force +- `b50ad87` 2026-07-14 — test-build: variant-namespace the dep cache and prune stale digests +- `4cf9fe4` 2026-07-14 — test-build: add --keep to save the built target package @@ -0,0 +1,338 @@ + GNU GENERAL PUBLIC LICENSE + Version 2, June 1991 + + Copyright (C) 1989, 1991 Free Software Foundation, Inc., + <https://fsf.org/> + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The licenses for most software are designed to take away your +freedom to share and change it. By contrast, the GNU General Public +License is intended to guarantee your freedom to share and change free +software--to make sure the software is free for all its users. This +General Public License applies to most of the Free Software +Foundation's software and to any other program whose authors commit to +using it. (Some other Free Software Foundation software is covered by +the GNU Lesser General Public License instead.) You can apply it to +your programs, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +this service if you wish), that you receive source code or can get it +if you want it, that you can change the software or use pieces of it +in new free programs; and that you know you can do these things. + + To protect your rights, we need to make restrictions that forbid +anyone to deny you these rights or to ask you to surrender the rights. +These restrictions translate to certain responsibilities for you if you +distribute copies of the software, or if you modify it. + + For example, if you distribute copies of such a program, whether +gratis or for a fee, you must give the recipients all the rights that +you have. You must make sure that they, too, receive or can get the +source code. And you must show them these terms so they know their +rights. + + We protect your rights with two steps: (1) copyright the software, and +(2) offer you this license which gives you legal permission to copy, +distribute and/or modify the software. + + Also, for each author's protection and ours, we want to make certain +that everyone understands that there is no warranty for this free +software. If the software is modified by someone else and passed on, we +want its recipients to know that what they have is not the original, so +that any problems introduced by others will not reflect on the original +authors' reputations. + + Finally, any free program is threatened constantly by software +patents. We wish to avoid the danger that redistributors of a free +program will individually obtain patent licenses, in effect making the +program proprietary. To prevent this, we have made it clear that any +patent must be licensed for everyone's free use or not licensed at all. + + The precise terms and conditions for copying, distribution and +modification follow. + + GNU GENERAL PUBLIC LICENSE + TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION + + 0. This License applies to any program or other work which contains +a notice placed by the copyright holder saying it may be distributed +under the terms of this General Public License. The "Program", below, +refers to any such program or work, and a "work based on the Program" +means either the Program or any derivative work under copyright law: +that is to say, a work containing the Program or a portion of it, +either verbatim or with modifications and/or translated into another +language. (Hereinafter, translation is included without limitation in +the term "modification".) Each licensee is addressed as "you". + +Activities other than copying, distribution and modification are not +covered by this License; they are outside its scope. The act of +running the Program is not restricted, and the output from the Program +is covered only if its contents constitute a work based on the +Program (independent of having been made by running the Program). +Whether that is true depends on what the Program does. + + 1. You may copy and distribute verbatim copies of the Program's +source code as you receive it, in any medium, provided that you +conspicuously and appropriately publish on each copy an appropriate +copyright notice and disclaimer of warranty; keep intact all the +notices that refer to this License and to the absence of any warranty; +and give any other recipients of the Program a copy of this License +along with the Program. + +You may charge a fee for the physical act of transferring a copy, and +you may at your option offer warranty protection in exchange for a fee. + + 2. You may modify your copy or copies of the Program or any portion +of it, thus forming a work based on the Program, and copy and +distribute such modifications or work under the terms of Section 1 +above, provided that you also meet all of these conditions: + + a) You must cause the modified files to carry prominent notices + stating that you changed the files and the date of any change. + + b) You must cause any work that you distribute or publish, that in + whole or in part contains or is derived from the Program or any + part thereof, to be licensed as a whole at no charge to all third + parties under the terms of this License. + + c) If the modified program normally reads commands interactively + when run, you must cause it, when started running for such + interactive use in the most ordinary way, to print or display an + announcement including an appropriate copyright notice and a + notice that there is no warranty (or else, saying that you provide + a warranty) and that users may redistribute the program under + these conditions, and telling the user how to view a copy of this + License. (Exception: if the Program itself is interactive but + does not normally print such an announcement, your work based on + the Program is not required to print an announcement.) + +These requirements apply to the modified work as a whole. If +identifiable sections of that work are not derived from the Program, +and can be reasonably considered independent and separate works in +themselves, then this License, and its terms, do not apply to those +sections when you distribute them as separate works. But when you +distribute the same sections as part of a whole which is a work based +on the Program, the distribution of the whole must be on the terms of +this License, whose permissions for other licensees extend to the +entire whole, and thus to each and every part regardless of who wrote it. + +Thus, it is not the intent of this section to claim rights or contest +your rights to work written entirely by you; rather, the intent is to +exercise the right to control the distribution of derivative or +collective works based on the Program. + +In addition, mere aggregation of another work not based on the Program +with the Program (or with a work based on the Program) on a volume of +a storage or distribution medium does not bring the other work under +the scope of this License. + + 3. You may copy and distribute the Program (or a work based on it, +under Section 2) in object code or executable form under the terms of +Sections 1 and 2 above provided that you also do one of the following: + + a) Accompany it with the complete corresponding machine-readable + source code, which must be distributed under the terms of Sections + 1 and 2 above on a medium customarily used for software interchange; or, + + b) Accompany it with a written offer, valid for at least three + years, to give any third party, for a charge no more than your + cost of physically performing source distribution, a complete + machine-readable copy of the corresponding source code, to be + distributed under the terms of Sections 1 and 2 above on a medium + customarily used for software interchange; or, + + c) Accompany it with the information you received as to the offer + to distribute corresponding source code. (This alternative is + allowed only for noncommercial distribution and only if you + received the program in object code or executable form with such + an offer, in accord with Subsection b above.) + +The source code for a work means the preferred form of the work for +making modifications to it. For an executable work, complete source +code means all the source code for all modules it contains, plus any +associated interface definition files, plus the scripts used to +control compilation and installation of the executable. However, as a +special exception, the source code distributed need not include +anything that is normally distributed (in either source or binary +form) with the major components (compiler, kernel, and so on) of the +operating system on which the executable runs, unless that component +itself accompanies the executable. + +If distribution of executable or object code is made by offering +access to copy from a designated place, then offering equivalent +access to copy the source code from the same place counts as +distribution of the source code, even though third parties are not +compelled to copy the source along with the object code. + + 4. You may not copy, modify, sublicense, or distribute the Program +except as expressly provided under this License. Any attempt +otherwise to copy, modify, sublicense or distribute the Program is +void, and will automatically terminate your rights under this License. +However, parties who have received copies, or rights, from you under +this License will not have their licenses terminated so long as such +parties remain in full compliance. + + 5. You are not required to accept this License, since you have not +signed it. However, nothing else grants you permission to modify or +distribute the Program or its derivative works. These actions are +prohibited by law if you do not accept this License. Therefore, by +modifying or distributing the Program (or any work based on the +Program), you indicate your acceptance of this License to do so, and +all its terms and conditions for copying, distributing or modifying +the Program or works based on it. + + 6. Each time you redistribute the Program (or any work based on the +Program), the recipient automatically receives a license from the +original licensor to copy, distribute or modify the Program subject to +these terms and conditions. You may not impose any further +restrictions on the recipients' exercise of the rights granted herein. +You are not responsible for enforcing compliance by third parties to +this License. + + 7. If, as a consequence of a court judgment or allegation of patent +infringement or for any other reason (not limited to patent issues), +conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot +distribute so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you +may not distribute the Program at all. For example, if a patent +license would not permit royalty-free redistribution of the Program by +all those who receive copies directly or indirectly through you, then +the only way you could satisfy both it and this License would be to +refrain entirely from distribution of the Program. + +If any portion of this section is held invalid or unenforceable under +any particular circumstance, the balance of the section is intended to +apply and the section as a whole is intended to apply in other +circumstances. + +It is not the purpose of this section to induce you to infringe any +patents or other property right claims or to contest validity of any +such claims; this section has the sole purpose of protecting the +integrity of the free software distribution system, which is +implemented by public license practices. Many people have made +generous contributions to the wide range of software distributed +through that system in reliance on consistent application of that +system; it is up to the author/donor to decide if he or she is willing +to distribute software through any other system and a licensee cannot +impose that choice. + +This section is intended to make thoroughly clear what is believed to +be a consequence of the rest of this License. + + 8. If the distribution and/or use of the Program is restricted in +certain countries either by patents or by copyrighted interfaces, the +original copyright holder who places the Program under this License +may add an explicit geographical distribution limitation excluding +those countries, so that distribution is permitted only in or among +countries not thus excluded. In such case, this License incorporates +the limitation as if written in the body of this License. + + 9. The Free Software Foundation may publish revised and/or new versions +of the General Public License from time to time. Such new versions will +be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + +Each version is given a distinguishing version number. If the Program +specifies a version number of this License which applies to it and "any +later version", you have the option of following the terms and conditions +either of that version or of any later version published by the Free +Software Foundation. If the Program does not specify a version number of +this License, you may choose any version ever published by the Free Software +Foundation. + + 10. If you wish to incorporate parts of the Program into other free +programs whose distribution conditions are different, write to the author +to ask for permission. For software which is copyrighted by the Free +Software Foundation, write to the Free Software Foundation; we sometimes +make exceptions for this. Our decision will be guided by the two goals +of preserving the free status of all derivatives of our free software and +of promoting the sharing and reuse of software generally. + + NO WARRANTY + + 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY +FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN +OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES +PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED +OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS +TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE +PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, +REPAIR OR CORRECTION. + + 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR +REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, +INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING +OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED +TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY +YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER +PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE +POSSIBILITY OF SUCH DAMAGES. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +convey the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + <one line to give the program's name and a brief idea of what it does.> + Copyright (C) <year> <name of author> + + This program is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 2 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License along + with this program; if not, see <https://www.gnu.org/licenses/>. + +Also add information on how to contact you by electronic and paper mail. + +If the program is interactive, make it output a short notice like this +when it starts in an interactive mode: + + Gnomovision version 69, Copyright (C) year name of author + Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. + This is free software, and you are welcome to redistribute it + under certain conditions; type `show c' for details. + +The hypothetical commands `show w' and `show c' should show the appropriate +parts of the General Public License. Of course, the commands you use may +be called something other than `show w' and `show c'; they could even be +mouse-clicks or menu items--whatever suits your program. + +You should also get your employer (if you work as a programmer) or your +school, if any, to sign a "copyright disclaimer" for the program, if +necessary. Here is a sample; alter the names: + + Yoyodyne, Inc., hereby disclaims all copyright interest in the program + `Gnomovision' (which makes passes at compilers) written by James Hacker. + + <signature of Moe Ghoul>, 1 April 1989 + Moe Ghoul, President of Vice + +This General Public License does not permit incorporating your program into +proprietary programs. If your program is a subroutine library, you may +consider it more useful to permit linking proprietary applications with the +library. If this is what you want to do, use the GNU Lesser General +Public License instead of this License. diff --git a/README.md b/README.md new file mode 100644 index 0000000..c52d4d2 --- /dev/null +++ b/README.md @@ -0,0 +1,85 @@ +# sbo-dockerbuild + +Docker-based toolchain for test-building SlackBuilds.org (SBo) packages on +Slackware, on both `-current` and `15.0`, without touching a real system. + +Two tools that share the same set of images: + +- **`test-build`** — verify a SlackBuild (the one you are editing, wherever it + lives) builds cleanly in a throwaway container. Resolves and builds its SBo + deps from a configured tree, caches built deps per image digest, lints the + target with `sbopkglint`, and reports per-package status. +- **`image-builder/`** — build the images `test-build` consumes + (`sbo-base` → `sbo-full` → `sbo-testbuild`, per Slackware version) from a + mirror of the Slackware package trees, pushed to a LAN registry. + +This started as a helper under `.extras/` in the **sbo-slackbuilds** package +repository and was extracted here to be maintained on its own. See +[HISTORY.md](HISTORY.md) for the original development log. + +## Layout + +``` +test-build the test-build tool +test-logic.sh pure-logic self-check for test-build (no docker) +test-build-config.example copy to ~/.config/sbo-testbuild/config and edit +overrides.example current-vs-stable dep overrides (optional) +image-builder/ the image-build scripts + their self-check +docs/specs, docs/plans design specs and implementation plans +``` + +## Install + +Copy the `test-build` CLI into `~/bin` (on your `$PATH`): + +```bash +./install.sh # -> ~/bin/test-build +BINDIR=/usr/local/bin ./install.sh +./install.sh --uninstall +``` + +It copies the file, so re-run after editing `test-build`. The `image-builder/` +scripts are not installed; they run on the docker host from their own checkout. + +## Quick start + +Test-build a package you are editing (from its dir, or by path): + +```bash +cd some-slackbuild-dir && test-build <name> # -current +test-build --stable <name> # 15.0 +test-build --keep <name> # also keep the built .txz +``` + +Build the images (on the docker host; see `image-builder/README`): + +```bash +cd image-builder +./bootstrap.sh --version current --force +./build-full-image.sh --version current --force +./build-sbo-testbuild.sh --version current --force +``` + +## Requirements + +- Docker with a reachable LAN registry for the built images. +- A local mirror of the Slackware package tree(s) for the image build, and a + checkout of the SBo tree(s) for dependency resolution. +- `sbo-maintainer-tools` (baked into the images for `sbopkglint`). + +See the specs under `docs/specs/` for the full design. + +## License + +Released under the GNU General Public License, version 2 (v2 only). See +[LICENSE](LICENSE). Each source file carries a short header notice. + +Copyright (C) 2026 Danilo M. <danix@danix.xyz> + +## Development Approach + +This project is developed using AI-assisted tools. Code is generated with the help of AI based on human-provided specifications, design decisions, and iterative feedback. + +All contributions are reviewed, tested, and curated by the maintainer before being included in the codebase. AI is used as a productivity and exploration tool, while human oversight remains central to all decisions. + +The goal is to combine the flexibility of AI-assisted development with standard open-source practices such as transparency, review, and accountability. diff --git a/docs/plans/2026-07-13-docker-test-build.md b/docs/plans/2026-07-13-docker-test-build.md new file mode 100644 index 0000000..6195de9 --- /dev/null +++ b/docs/plans/2026-07-13-docker-test-build.md @@ -0,0 +1,1579 @@ +# Docker test-build Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** A single bash script `.extras/test-build` that verifies an already-published SBo package still builds cleanly on `-current` or `15.0` inside a throwaway docker container, resolving and building its SBo deps from the local tree, caching built deps per image digest, and reporting per-package status + lint. + +**Architecture:** Host resolves the dep tree from the local SBo tree (logic ported from `sbo-batch-test`), applies current-vs-stable override rules, prints the build order and confirms, then runs one `docker run --rm` that builds deps (or installs them from a host cache) and the target, lints the target, and streams results back. The container is the disposable environment (no overlayfs). Images are built by a separate job and consumed by tag. + +**Tech Stack:** Bash, docker, Slackware `installpkg`/`makepkg`, `sbo-maintainer-tools` (`sbopkglint`). Self-check is plain bash (`test-logic.sh`), no framework. + +--- + +## Reference source + +Port from `/home/danix/Programming/GIT/sbo-batch-tester/sbo-batch-test`. These +functions are lifted with minimal change (they are pure or host-side): + +- `find_slackbuild_dir`, `category_of`, `pkg_key`, `read_requires`, + `version_of`, `installed_in_base` +- resolver: `_resolve_visit`, `resolve_target`, globals `RESOLVED_ORDER`, + `UNMET`, `CYCLES`, `HAS_README`, `_vstate` +- cache: `_cache_ver_of`, `cache_decision`, `cache_path`, `cache_store`, + `cache_label` +- `depends_on_failed`, `print_summary`, `lint_pkg`, `init_color` +- self-check patterns from + `/home/danix/Programming/GIT/sbo-batch-tester/test-logic.sh` + +What is NEW or CHANGED in this repo: + +- overlay lifecycle (`setup_overlay`/`teardown_overlay`/`cleanup_trap`) is + REMOVED. The container replaces it. +- `build_one` is rewritten to run its build steps via `docker run` instead of + `chroot`. +- image + tree selection by version (`--stable`). +- override engine (drop / rename / fetch) applied to the resolved order. +- cache namespaced by image digest. +- config keys differ (see Task 2). + +## File Structure + +- Create: `/home/danix/Programming/GIT/sbo-slackbuilds/.extras/test-build` + (the script, executable) +- Create: + `/home/danix/Programming/GIT/sbo-slackbuilds/.extras/test-build-config.example` + (config template) +- Create: + `/home/danix/Programming/GIT/sbo-slackbuilds/.extras/overrides.example` + (override-rules template) +- Create: + `/home/danix/Programming/GIT/sbo-slackbuilds/.extras/test-logic.sh` + (self-check) + +All work happens in `.extras/`. Run every command from the repo root +`/home/danix/Programming/GIT/sbo-slackbuilds`. + +Note on committing: this repo has a pre-commit hook that runs `sbolint` on +staged packages. `.extras/` files are not packages, but to be safe every +commit in this plan uses `SBOLINT=no git commit ...`. Commits are GPG-signed +automatically; do not disable signing. + +--- + +## Task 1: Scaffold the script + smoke self-check + +**Files:** +- Create: `.extras/test-build` +- Create: `.extras/test-logic.sh` + +- [ ] **Step 1: Write the failing self-check** + +Create `.extras/test-logic.sh`: + +```bash +#!/bin/bash +# +# Logic self-check for .extras/test-build. Covers the pure, host-side parts: +# dependency resolution, override application, unknown-dep -> UNMET, cache +# decisions, and BLOCKED-BY-DEP propagation. No docker. +# +# Run: bash .extras/test-logic.sh +# +set -uo pipefail + +SCRIPT="$(dirname "$0")/test-build" +T=$(mktemp -d) +BASE_DB=$(mktemp -d); mkdir -p "$BASE_DB" + +cleanup() { rm -rf "$T" "$BASE_DB"; } +trap cleanup EXIT + +# Source the script without running main(). Sourcing re-runs the CONFIG block, +# so set test vars AFTER the source. +LIB=$(mktemp) +sed '/^main "\$@"$/d' "$SCRIPT" > "$LIB" +# shellcheck disable=SC1090 +source "$LIB" 2>/dev/null +rm -f "$LIB" + +pass=0; fail=0 +ok() { echo " ok: $1"; ((pass++)); return 0; } +bad() { echo " FAIL: $1"; ((fail++)); return 0; } + +# Placeholder assertion so this file runs before any logic exists. +ok "script sources without executing main" + +echo +echo "$pass passed, $fail failed" +[[ $fail -eq 0 ]] || exit 1 +echo "ALL LOGIC CHECKS PASS" +``` + +- [ ] **Step 2: Run it to verify it fails** + +Run: `bash .extras/test-logic.sh` +Expected: FAIL. `$SCRIPT` does not exist yet, `sed`/`source` on a missing file +errors, no "ALL LOGIC CHECKS PASS". + +- [ ] **Step 3: Write the minimal script skeleton** + +Create `.extras/test-build`: + +```bash +#!/bin/bash +# +# test-build - verify an already-published SBo package still builds on a target +# Slackware version inside a throwaway docker container. Resolves + builds its +# SBo deps from the local tree, caches built deps per image digest, reports +# per-package status and lints the target. +# +# Dependency-resolution, cache, and summary logic are adapted from sbo-batch-test +# (github: danixland). The overlay chroot is replaced by a docker container: the +# container IS the disposable environment, so no overlayfs. +# +# No em dashes in prose by author convention. + +# ============================================================================= +# CONFIG (do not edit here; real values live in the external config file) +# ============================================================================= +SBO_TREE_CURRENT="" +SBO_TREE_STABLE="" +IMAGE_CURRENT="" +IMAGE_STABLE="" +LOG_ROOT="/var/log/sbo-test-build" +PKG_CACHE="" + +TB_CONFIG="${TB_CONFIG:-$HOME/.config/sbo-testbuild/config}" +if [[ -f "$TB_CONFIG" ]]; then + # shellcheck disable=SC1090 + source "$TB_CONFIG" +fi + +TB_OVERRIDES="${TB_OVERRIDES:-$HOME/.config/sbo-testbuild/overrides}" + +# ============================================================================= +set -uo pipefail +# Not -e: a package build failing is a handled outcome, not a script crash. + +main() { + : +} + +main "$@" +``` + +- [ ] **Step 4: Run the self-check to verify it passes** + +Run: `bash .extras/test-logic.sh` +Expected: PASS, ending `ALL LOGIC CHECKS PASS`. + +- [ ] **Step 5: Commit** + +```bash +chmod +x .extras/test-build +git add .extras/test-build .extras/test-logic.sh +SBOLINT=no git commit -m 'test-build: scaffold script + self-check' +``` + +--- + +## Task 2: Config, arg parsing, version selection + +**Files:** +- Modify: `.extras/test-build` +- Create: `.extras/test-build-config.example` +- Test: `.extras/test-logic.sh` + +- [ ] **Step 1: Write the failing test** + +Append to `.extras/test-logic.sh` before the `--- result ---` block (the +`echo; "$pass passed..."` lines at the end): + +```bash +# --- version selection ------------------------------------------------------ +SBO_TREE_CURRENT="/trees/current" +SBO_TREE_STABLE="/trees/stable" +IMAGE_CURRENT="sbo-testbuild:current" +IMAGE_STABLE="sbo-testbuild:15.0" + +VERSION_ID="current"; select_version_paths +[[ "$ACTIVE_TREE" == "/trees/current" ]] && ok "current -> current tree" || bad "current tree wrong: [$ACTIVE_TREE]" +[[ "$ACTIVE_IMAGE" == "sbo-testbuild:current" ]] && ok "current -> current image" || bad "current image wrong: [$ACTIVE_IMAGE]" + +VERSION_ID="15.0"; select_version_paths +[[ "$ACTIVE_TREE" == "/trees/stable" ]] && ok "15.0 -> stable tree" || bad "stable tree wrong: [$ACTIVE_TREE]" +[[ "$ACTIVE_IMAGE" == "sbo-testbuild:15.0" ]] && ok "15.0 -> stable image" || bad "stable image wrong: [$ACTIVE_IMAGE]" +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `bash .extras/test-logic.sh` +Expected: FAIL, `select_version_paths: command not found`. + +- [ ] **Step 3: Implement config globals, flags, parse_args, select_version_paths** + +In `.extras/test-build`, replace the `main() { : }` line and everything up to +`main "$@"` with the flags/globals and functions below (keep the CONFIG block +above untouched): + +```bash +# ---- flags / globals -------------------------------------------------------- +USE_COLOR=1 # --no-color or non-TTY disables +DRY_RUN=0 # --dry-run: resolve + print order, do not build +ASSUME_YES=0 # --yes: skip the confirm prompt (still prints the order) +USE_CACHE=1 # --no-cache disables the dep cache for one run +VERSION_ID="current" # "current" | "15.0"; set by --stable +TARGET_ARG="" + +ACTIVE_TREE="" # selected SBo tree (by version) +ACTIVE_IMAGE="" # selected image tag (by version) +RUN_DIR="" # timestamped log dir for this run + +# Status tracking. Keyed by "category/prog". Parallel assoc arrays. +declare -A ST_STATUS=() +declare -A ST_REASON=() +declare -A ST_TIME=() +declare -A ST_README=() + +usage() { + cat <<'EOF' +test-build - verify an SBo package builds on a target Slackware in docker + +USAGE: + test-build [OPTIONS] <program-name> + +OPTIONS: + -h, --help This text. + --stable Target Slackware 15.0 (image + tree). Default is -current. + --dry-run Resolve, apply overrides, print the build order, do not build. + --yes Skip the confirm prompt (the order is still printed first). + --no-cache Rebuild all deps this run, ignore/refresh the cache. + --no-color Disable ANSI color (auto-disabled when stdout is not a TTY). +EOF +} + +parse_args() { + while [[ $# -gt 0 ]]; do + case "$1" in + -h|--help) usage; exit 0 ;; + --stable|15.0) VERSION_ID="15.0"; shift ;; + --dry-run) DRY_RUN=1; shift ;; + --yes) ASSUME_YES=1; shift ;; + --no-cache) USE_CACHE=0; shift ;; + --no-color) USE_COLOR=0; shift ;; + -*) echo "Unknown option: $1" >&2; usage >&2; exit 2 ;; + *) + if [[ -n "$TARGET_ARG" ]]; then + echo "Only one target accepted (got '$TARGET_ARG' and '$1')." >&2 + exit 2 + fi + TARGET_ARG="$1"; shift ;; + esac + done + if [[ -z "$TARGET_ARG" ]]; then + echo "No target given." >&2; usage >&2; exit 2 + fi +} + +init_color() { + if [[ $USE_COLOR -eq 1 && -t 1 ]]; then + C_RED=$'\e[31m'; C_GRN=$'\e[32m'; C_YEL=$'\e[33m'; C_RST=$'\e[0m' + else + C_RED=""; C_GRN=""; C_YEL=""; C_RST="" + fi +} + +# Map VERSION_ID to the active tree + image. No I/O, unit-testable. +select_version_paths() { + if [[ "$VERSION_ID" == "15.0" ]]; then + ACTIVE_TREE="$SBO_TREE_STABLE"; ACTIVE_IMAGE="$IMAGE_STABLE" + else + ACTIVE_TREE="$SBO_TREE_CURRENT"; ACTIVE_IMAGE="$IMAGE_CURRENT" + fi +} + +# require_config: the external config must exist and set the version's paths. +require_config() { + if [[ ! -f "$TB_CONFIG" ]]; then + cat >&2 <<EOF +No config file: $TB_CONFIG +Copy the example and edit it: + + mkdir -p "\$(dirname "$TB_CONFIG")" + cp .extras/test-build-config.example "$TB_CONFIG" + \${EDITOR:-vi} "$TB_CONFIG" +EOF + exit 1 + fi + select_version_paths + if [[ -z "$ACTIVE_TREE" || -z "$ACTIVE_IMAGE" ]]; then + echo "Config $TB_CONFIG is missing the tree/image for version '$VERSION_ID'." >&2 + exit 1 + fi + if [[ ! -d "$ACTIVE_TREE" ]]; then + echo "SBo tree for '$VERSION_ID' does not exist: $ACTIVE_TREE" >&2 + exit 1 + fi +} + +main() { + parse_args "$@" + init_color + require_config +} + +main "$@" +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `bash .extras/test-logic.sh` +Expected: PASS (the new version-selection checks included). + +- [ ] **Step 5: Write the config example** + +Create `.extras/test-build-config.example`: + +```bash +# test-build config. Copy to ~/.config/sbo-testbuild/config and edit. +# Override the path with the TB_CONFIG environment variable. + +# Local SBo trees, one per target Slackware version. +SBO_TREE_CURRENT="/home/danix/SBo/current" +SBO_TREE_STABLE="/home/danix/SBo/15.0" + +# Ready image tags (built by a separate job, consumed here by tag). +IMAGE_CURRENT="sbo-testbuild:current" +IMAGE_STABLE="sbo-testbuild:15.0" + +# Where per-run logs land. +LOG_ROOT="/home/danix/.cache/sbo-test-build" + +# Dependency package cache (host dir). Empty disables caching. +# Namespaced internally by image digest, so it self-invalidates on image update +# and keeps current vs 15.0 separate. +PKG_CACHE="/home/danix/.cache/sbo-test-build/pkgcache" +``` + +- [ ] **Step 6: Commit** + +```bash +git add .extras/test-build .extras/test-build-config.example .extras/test-logic.sh +SBOLINT=no git commit -m 'test-build: config, arg parsing, version selection' +``` + +--- + +## Task 3: Port the SBo-tree lookup + resolver + self-check + +**Files:** +- Modify: `.extras/test-build` +- Test: `.extras/test-logic.sh` + +- [ ] **Step 1: Write the failing test** + +Append to `.extras/test-logic.sh` before the result block: + +```bash +# --- resolution ------------------------------------------------------------- +# Fake SBo tree under one category. mk <prog> "<REQUIRES>". +mk() { mkdir -p "$T/cat/$1"; echo "REQUIRES=\"$2\"" > "$T/cat/$1/$1.info"; } +mk c "" +mk b "c" +mk a "b %README%" +mk d "nonexistentpkg" +mk e "f" +mk f "e" +mk g "b" + +SBO_TREE_ROOTS=("$T") # resolver reads this global (set by select in real runs) + +resolve_target "$T/cat/a" +order=""; for x in "${RESOLVED_ORDER[@]}"; do order+="$(basename "$x") "; done +order="${order% }" +[[ "$order" == "c b a" ]] && ok "topo order c b a" || bad "topo order, got: [$order]" +[[ "${HAS_README[$T/cat/a]:-}" == "1" ]] && ok "%README% recorded" || bad "%README% not recorded" +[[ ${#UNMET[@]} -eq 0 ]] && ok "no false unmet" || bad "unexpected unmet" + +resolve_target "$T/cat/d" +[[ ${#UNMET[@]} -eq 1 ]] && ok "unmet-dep caught" || bad "unmet-dep missed" + +resolve_target "$T/cat/e" +[[ ${#CYCLES[@]} -ge 1 ]] && ok "cycle caught" || bad "cycle missed" +``` + +Note: the base-installed check (`installed_in_base`) is exercised in real runs +against the container, not here. In the self-check `BASE_DB` is empty, so a dep +that is neither in the tree nor a known override reads as UNMET (which is the +behavior we test with `d`). + +- [ ] **Step 2: Run test to verify it fails** + +Run: `bash .extras/test-logic.sh` +Expected: FAIL, `resolve_target: command not found`. + +- [ ] **Step 3: Implement lookup + resolver (ported)** + +Add to `.extras/test-build` above `main()`: + +```bash +# ============================================================================= +# SBo tree lookup +# ============================================================================= +# In this tool there is one active tree per run, but the resolver reads an array +# named SBO_TREE_ROOTS so the ported logic and its self-check match sbo-batch-test. +# main() sets SBO_TREE_ROOTS=("$ACTIVE_TREE") after require_config. +declare -a SBO_TREE_ROOTS=() + +find_slackbuild_dir() { + local prog="$1" root d + for root in "${SBO_TREE_ROOTS[@]}"; do + [[ -d "$root" ]] || continue + for d in "$root"/*/"$prog"; do + if [[ -d "$d" && -f "$d/$prog.info" ]]; then + echo "$d"; return 0 + fi + done + done + return 1 +} + +category_of() { basename "$(dirname "$1")"; } +pkg_key() { echo "$(category_of "$1")/$(basename "$1")"; } + +read_requires() { + local info="$1" + # shellcheck disable=SC1090 + ( set +u; source "$info"; echo "${REQUIRES:-}" ) +} + +version_of() { + local dir="$1" + local info="$dir/$(basename "$dir").info" + [[ -f "$info" ]] || return + local v; v="$(grep -m1 '^VERSION=' "$info" | cut -d'"' -f2)" + echo "$v" +} + +# ============================================================================= +# DEPENDENCY RESOLUTION (topo sort + cycle detection, LOCAL tree only) +# ============================================================================= +declare -a RESOLVED_ORDER=() +declare -A UNMET=() +declare -a CYCLES=() +declare -A HAS_README=() +declare -A _vstate=() + +# Is a prog already present in the container base? Overridden at build time to +# consult the image; in resolution we treat "in base" as a callback so the pure +# topo logic stays testable. Default: not in base (self-check has no base db). +installed_in_base() { return 1; } + +_resolve_visit() { + local dir="$1" parent="$2" + local key; key="$(basename "$dir")" + + if [[ "${_vstate[$dir]:-}" == "1" ]]; then return 0; fi + if [[ "${_vstate[$dir]:-}" == "0" ]]; then + CYCLES+=("cycle involving $key (pulled in via $parent)") + return 1 + fi + _vstate["$dir"]=0 + + local info="$dir/$(basename "$dir").info" + local req tok depdir rc=0 + req="$(read_requires "$info")" + for tok in $req; do + if [[ "$tok" == "%README%" ]]; then + HAS_README["$dir"]=1 + continue + fi + if depdir="$(find_slackbuild_dir "$tok")"; then + _resolve_visit "$depdir" "$key" || rc=1 + elif installed_in_base "$tok"; then + : + else + UNMET["$tok"]="needed by $key" + rc=1 + fi + done + + _vstate["$dir"]=1 + RESOLVED_ORDER+=("$dir") + return $rc +} + +resolve_target() { + local dir="$1" + RESOLVED_ORDER=() + CYCLES=() + UNMET=() + _vstate=() + _resolve_visit "$dir" "(top)" +} +``` + +Update `main()` to set the resolver's tree array after `require_config`: + +```bash +main() { + parse_args "$@" + init_color + require_config + SBO_TREE_ROOTS=("$ACTIVE_TREE") +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `bash .extras/test-logic.sh` +Expected: PASS (topo order, %README%, unmet, cycle checks green). + +- [ ] **Step 5: Commit** + +```bash +git add .extras/test-build .extras/test-logic.sh +SBOLINT=no git commit -m 'test-build: port SBo lookup + dependency resolver' +``` + +--- + +## Task 4: Override engine (drop / rename / fetch) + UNMET stop + +**Files:** +- Modify: `.extras/test-build` +- Create: `.extras/overrides.example` +- Test: `.extras/test-logic.sh` + +The override file encodes current-vs-stable deltas. It is applied ONLY when +`VERSION_ID == current` (15.0 is the SBo baseline). Rules: + +- `drop: <prog>` remove from the build order (already in -current base) +- `rename: <old> -> <new>` rewrite the dep name before it is looked up +- `fetch: <prog>` mark this dep to come from sbopkg in-container, not the local tree + +- [ ] **Step 1: Write the failing test** + +Append to `.extras/test-logic.sh` before the result block: + +```bash +# --- overrides -------------------------------------------------------------- +OV=$(mktemp) +cat > "$OV" <<'EOF' +# comment ignored +drop: dropme +rename: oldname -> newname +fetch: fetchme +EOF +TB_OVERRIDES="$OV" +load_overrides + +[[ "${OV_DROP[dropme]:-}" == "1" ]] && ok "override drop parsed" || bad "drop not parsed" +[[ "${OV_RENAME[oldname]:-}" == "newname" ]] && ok "override rename parsed" || bad "rename not parsed" +[[ "${OV_FETCH[fetchme]:-}" == "1" ]] && ok "override fetch parsed" || bad "fetch not parsed" + +# rename maps a token +[[ "$(apply_rename oldname)" == "newname" ]] && ok "apply_rename maps" || bad "apply_rename wrong: [$(apply_rename oldname)]" +[[ "$(apply_rename untouched)" == "untouched" ]] && ok "apply_rename passthrough" || bad "apply_rename mangled untouched" + +# drop filters the order (build a fake order of dirs; dropme is removed) +mk dropme "" +mk keepme "" +RESOLVED_ORDER=("$T/cat/dropme" "$T/cat/keepme") +apply_overrides_to_order +kept=""; for x in "${RESOLVED_ORDER[@]}"; do kept+="$(basename "$x") "; done +kept="${kept% }" +[[ "$kept" == "keepme" ]] && ok "drop removes from order" || bad "drop failed, order=[$kept]" +rm -f "$OV" + +# overrides only apply on current; on 15.0 they are a no-op +VERSION_ID="15.0" +TB_OVERRIDES="$OV" # file gone; must not error, must clear maps +load_overrides +[[ ${#OV_DROP[@]} -eq 0 ]] && ok "overrides inert on 15.0" || bad "overrides applied on 15.0" +VERSION_ID="current" +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `bash .extras/test-logic.sh` +Expected: FAIL, `load_overrides: command not found`. + +- [ ] **Step 3: Implement the override engine** + +Add to `.extras/test-build` above `main()`: + +```bash +# ============================================================================= +# current-vs-stable overrides. Parsed from $TB_OVERRIDES. Applied only when +# targeting -current (15.0 is the SBo baseline, no deltas). +# ============================================================================= +declare -A OV_DROP=() # prog -> 1 +declare -A OV_RENAME=() # old -> new +declare -A OV_FETCH=() # prog -> 1 + +load_overrides() { + OV_DROP=(); OV_RENAME=(); OV_FETCH=() + # 15.0 is the baseline: no overrides. + [[ "$VERSION_ID" == "15.0" ]] && return + [[ -f "$TB_OVERRIDES" ]] || return + local line kind rest + while IFS= read -r line; do + line="${line%%#*}" # strip comments + line="${line#"${line%%[![:space:]]*}"}" # ltrim + [[ -z "$line" ]] && continue + kind="${line%%:*}"; rest="${line#*:}" + kind="${kind//[[:space:]]/}" + rest="${rest#"${rest%%[![:space:]]*}"}" # ltrim value + case "$kind" in + drop) OV_DROP["${rest//[[:space:]]/}"]=1 ;; + fetch) OV_FETCH["${rest//[[:space:]]/}"]=1 ;; + rename) + # rest is "old -> new" + local old new + old="${rest%%->*}"; new="${rest##*->}" + old="${old//[[:space:]]/}"; new="${new//[[:space:]]/}" + [[ -n "$old" && -n "$new" ]] && OV_RENAME["$old"]="$new" ;; + *) echo "WARN: unknown override rule: $line" >&2 ;; + esac + done < "$TB_OVERRIDES" +} + +# Map a dep token through rename rules (identity if no rule). +apply_rename() { + local tok="$1" + echo "${OV_RENAME[$tok]:-$tok}" +} + +# Remove dropped packages from RESOLVED_ORDER in place. +apply_overrides_to_order() { + local d prog keep=() + for d in "${RESOLVED_ORDER[@]}"; do + prog="$(basename "$d")" + [[ "${OV_DROP[$prog]:-}" == "1" ]] && continue + keep+=("$d") + done + RESOLVED_ORDER=("${keep[@]:-}") +} +``` + +Wire `apply_rename` into the resolver so renamed deps look up under the new +name. In `_resolve_visit`, change the dep-lookup line: + +```bash + for tok in $req; do + if [[ "$tok" == "%README%" ]]; then + HAS_README["$dir"]=1 + continue + fi + tok="$(apply_rename "$tok")" # <-- add this line + if [[ "${OV_FETCH[$tok]:-}" == "1" ]]; then + FETCH_DEPS["$tok"]=1 # satisfied via sbopkg at build time + continue + fi + if depdir="$(find_slackbuild_dir "$tok")"; then + _resolve_visit "$depdir" "$key" || rc=1 + elif installed_in_base "$tok"; then + : + else + UNMET["$tok"]="needed by $key" + rc=1 + fi + done +``` + +Add the `FETCH_DEPS` global with the other resolver globals: + +```bash +declare -A FETCH_DEPS=() # prog -> 1: resolve via sbopkg in-container +``` + +And reset it in `resolve_target`: + +```bash +resolve_target() { + local dir="$1" + RESOLVED_ORDER=() + CYCLES=() + UNMET=() + FETCH_DEPS=() + _vstate=() + _resolve_visit "$dir" "(top)" +} +``` + +Load overrides in `main()` (before any resolve happens, after tree setup): + +```bash +main() { + parse_args "$@" + init_color + require_config + SBO_TREE_ROOTS=("$ACTIVE_TREE") + load_overrides +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `bash .extras/test-logic.sh` +Expected: PASS (all override checks green). + +- [ ] **Step 5: Write the overrides example** + +Create `.extras/overrides.example`: + +``` +# current-vs-stable dep overrides for test-build. +# Copy to ~/.config/sbo-testbuild/overrides and edit. Applied only when +# targeting -current (15.0 is the SBo baseline). One rule per line. +# +# drop: <prog> dep is in 15.0 but already in the -current base +# rename: <old> -> <new> dep was renamed on -current +# fetch: <prog> dep was removed from the -current tree; pull via sbopkg +# +# Start light: add rules only when a real package needs them. + +# drop: rust +# rename: python3-foo -> foo +# fetch: somelib +``` + +- [ ] **Step 6: Commit** + +```bash +git add .extras/test-build .extras/overrides.example .extras/test-logic.sh +SBOLINT=no git commit -m 'test-build: current-vs-stable override engine' +``` + +--- + +## Task 5: Port the dep cache, namespaced by image digest + +**Files:** +- Modify: `.extras/test-build` +- Test: `.extras/test-logic.sh` + +The cache is the ported sbo-batch-test cache with one change: the cache root +seen by `cache_*` is `$PKG_CACHE/<image-digest>`, computed once per run. A new +digest gives a fresh namespace (old deps ignored), which is the invalidation. + +- [ ] **Step 1: Write the failing test** + +Append to `.extras/test-logic.sh` before the result block: + +```bash +# --- cache ------------------------------------------------------------------ +PKG_CACHE=$(mktemp -d) +CACHE_ROOT="$PKG_CACHE/sha256-deadbeef" # simulate a resolved digest namespace +mkc() { mkdir -p "$CACHE_ROOT/$1/$2"; : > "$CACHE_ROOT/$1/$2/$3"; } + +mkc net libfoo "libfoo-1.1-x86_64-1_danix.txz" +[[ "$(cache_decision net libfoo 1.1)" == "cached" ]] && ok "cache hit on version match" || bad "cache_decision got [$(cache_decision net libfoo 1.1)]" +[[ "$(cache_decision net libfoo 1.2)" == "bump:1.1:1.2" ]] && ok "cache bump reported" || bad "cache bump got [$(cache_decision net libfoo 1.2)]" +[[ "$(cache_decision net libbar 1.0)" == "new" ]] && ok "cache new for absent" || bad "cache new got [$(cache_decision net libbar 1.0)]" + +hit="$(cache_path net libfoo 1.1)" +[[ "$hit" == "$CACHE_ROOT/net/libfoo/libfoo-1.1-x86_64-1_danix.txz" ]] && ok "cache_path returns hit" || bad "cache_path got [$hit]" +[[ -z "$(cache_path net libfoo 9.9)" ]] && ok "cache_path empty on miss" || bad "cache_path not empty on miss" + +srctmp=$(mktemp -d); : > "$srctmp/libfoo-1.2-x86_64-1_danix.txz" +cache_store net libfoo "$srctmp/libfoo-1.2-x86_64-1_danix.txz" +count=$(find "$CACHE_ROOT/net/libfoo" -name '*.t?z' | wc -l) +[[ "$count" -eq 1 ]] && ok "cache_store evicts to one file" || bad "cache_store left $count files" +[[ -e "$CACHE_ROOT/net/libfoo/libfoo-1.2-x86_64-1_danix.txz" ]] && ok "cache_store stored new file" || bad "cache_store did not store" +rm -rf "$srctmp" + +# disabled cache -> new +CACHE_ROOT_SAVE="$CACHE_ROOT"; USE_CACHE=0 +[[ "$(cache_decision net libfoo 1.2)" == "new" ]] && ok "--no-cache disables (new)" || bad "disabled cache got [$(cache_decision net libfoo 1.2)]" +USE_CACHE=1; CACHE_ROOT="$CACHE_ROOT_SAVE" +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `bash .extras/test-logic.sh` +Expected: FAIL, `cache_decision: command not found`. + +- [ ] **Step 3: Implement the cache (ported, keyed on CACHE_ROOT + USE_CACHE)** + +Add to `.extras/test-build` above `main()`: + +```bash +# ============================================================================= +# Dependency cache. Layout: $CACHE_ROOT/<cat>/<prog>/<prog>-<ver>-...txz where +# CACHE_ROOT = $PKG_CACHE/<image-digest> (set per run by resolve_cache_root). +# Key is prog+version. --no-cache (USE_CACHE=0) or empty PKG_CACHE disables. +# ============================================================================= +CACHE_ROOT="" # set by resolve_cache_root once the image digest is known + +# True when the cache is usable this run. +_cache_on() { [[ $USE_CACHE -eq 1 && -n "$PKG_CACHE" && -n "$CACHE_ROOT" ]]; } + +_cache_ver_of() { + local prog="$1" base="$2" + base="${base#"$prog"-}" + echo "${base%%-*}" +} + +# cache_decision <cat> <prog> <version> -> cached | bump:OLD:NEW | new +cache_decision() { + local cat="$1" prog="$2" version="$3" + _cache_on || { echo new; return; } + local dir="$CACHE_ROOT/$cat/$prog" + local f newest="" + for f in "$dir/$prog"-*.t?z; do + [[ -e "$f" ]] || continue + [[ -z "$newest" || "$f" -nt "$newest" ]] && newest="$f" + done + [[ -z "$newest" ]] && { echo new; return; } + local have; have="$(_cache_ver_of "$prog" "$(basename "$newest")")" + if [[ "$have" == "$version" ]]; then echo cached; else echo "bump:$have:$version"; fi +} + +cache_path() { + local cat="$1" prog="$2" version="$3" + _cache_on || return + local dir="$CACHE_ROOT/$cat/$prog" + local f newest="" + for f in "$dir/$prog"-*.t?z; do + [[ -e "$f" ]] || continue + [[ -z "$newest" || "$f" -nt "$newest" ]] && newest="$f" + done + [[ -z "$newest" ]] && return + [[ "$(_cache_ver_of "$prog" "$(basename "$newest")")" == "$version" ]] && echo "$newest" +} + +cache_store() { + local cat="$1" prog="$2" src="$3" + _cache_on || return + local dir="$CACHE_ROOT/$cat/$prog" + mkdir -p "$dir" + rm -f "$dir"/*.t?z + cp -a "$src" "$dir/" +} + +cache_label() { + local dir="$1" is_target="$2" + local cat prog ver dec + cat="$(category_of "$dir")"; prog="$(basename "$dir")"; ver="$(version_of "$dir")" + dec="$(cache_decision "$cat" "$prog" "$ver")" + local label + case "$dec" in + cached) label="cached ($ver)" ;; + bump:*) label="rebuild: ${dec#bump:}"; label="${label/:/ -> }" ;; + *) label="build (new)" ;; + esac + if [[ "$is_target" == "1" ]]; then + case "$dec" in + cached) label="build (cached $ver, rebuilt as target)" ;; + esac + echo "target, $label" + else + echo "$label" + fi +} + +# Compute CACHE_ROOT from the image's digest. Falls back to the tag if the +# digest cannot be read (still isolates per image reference). +resolve_cache_root() { + [[ -z "$PKG_CACHE" ]] && { CACHE_ROOT=""; return; } + local digest + digest="$(docker image inspect --format '{{index .Id}}' "$ACTIVE_IMAGE" 2>/dev/null)" + [[ -z "$digest" ]] && digest="tag-${ACTIVE_IMAGE//[^a-zA-Z0-9._-]/_}" + digest="${digest//[^a-zA-Z0-9._-]/_}" + CACHE_ROOT="$PKG_CACHE/$digest" + mkdir -p "$CACHE_ROOT" +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `bash .extras/test-logic.sh` +Expected: PASS (all cache checks green). + +- [ ] **Step 5: Commit** + +```bash +git add .extras/test-build .extras/test-logic.sh +SBOLINT=no git commit -m 'test-build: port dep cache, namespaced by image digest' +``` + +--- + +## Task 6: BLOCKED-BY-DEP propagation (ported) + +**Files:** +- Modify: `.extras/test-build` +- Test: `.extras/test-logic.sh` + +- [ ] **Step 1: Write the failing test** + +Append to `.extras/test-logic.sh` before the result block: + +```bash +# --- BLOCKED-BY-DEP (depends_on_failed) ------------------------------------- +dead=(b) +if depends_on_failed "$T/cat/g" dead; then ok "g blocked when b dead"; else bad "g should block on b"; fi +if depends_on_failed "$T/cat/a" dead; then ok "a blocked when b dead (direct)"; else bad "a should block on b"; fi + +dead=(c) +if depends_on_failed "$T/cat/a" dead; then bad "a wrongly blocked on c"; else ok "a not directly blocked by c"; fi +if depends_on_failed "$T/cat/b" dead; then ok "b blocked when c dead"; else bad "b should block on c"; fi + +dead=() +if depends_on_failed "$T/cat/a" dead; then bad "a blocked with empty dead"; else ok "no block when nothing dead"; fi + +dead=("%README%") +if depends_on_failed "$T/cat/a" dead; then bad "%README% treated as dep"; else ok "%README% not treated as dep"; fi +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `bash .extras/test-logic.sh` +Expected: FAIL, `depends_on_failed: command not found`. + +- [ ] **Step 3: Implement depends_on_failed (ported)** + +Add to `.extras/test-build` above `main()`: + +```bash +# Does SlackBuild dir $1 directly require any prog in the dead list (nameref $2)? +# Direct-requires check only; transitive blocking works because run_target +# iterates in topo order, propagating a failure one hop per package. +depends_on_failed() { + local dir="$1"; local -n failed="$2" + local info="$dir/$(basename "$dir").info" + local req tok f + req="$(read_requires "$info")" + for tok in $req; do + [[ "$tok" == "%README%" ]] && continue + for f in "${failed[@]:-}"; do + [[ "$tok" == "$f" ]] && return 0 + done + done + return 1 +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `bash .extras/test-logic.sh` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add .extras/test-build .extras/test-logic.sh +SBOLINT=no git commit -m 'test-build: port BLOCKED-BY-DEP propagation' +``` + +--- + +## Task 7: The container build step (build_one via docker) + +**Files:** +- Modify: `.extras/test-build` + +No self-check here (docker/build is out of the pure-logic reach, same boundary +as sbo-batch-test). Verified by a real run in Task 10. + +- [ ] **Step 1: Implement build_one + lint_pkg** + +Add to `.extras/test-build` above `main()`. + +`lint_pkg` is ported verbatim: + +```bash +# lint_pkg <txz> <logf> -> run sbopkglint on a built package. Host-side, target +# only, fail-soft (skip if absent, never change SUCCESS). +lint_pkg() { + local txz="$1" logf="$2" + if ! command -v sbopkglint >/dev/null 2>&1; then + echo " sbopkglint not installed on host, skipping lint" + return 0 + fi + echo " sbopkglint $(basename "$txz") ..." + echo "===== sbopkglint: $(basename "$txz") =====" >> "$logf" + local out rc + out="$(sbopkglint "$txz" 2>&1)"; rc=$? + printf '%s\n' "$out" >> "$logf" + if [[ $rc -eq 0 ]]; then + echo " lint: ${C_GRN}clean${C_RST}" + else + echo " lint: ${C_RED}findings${C_RST} (see $(basename "$logf")):" + printf '%s\n' "$out" | sed 's/^/ /' + fi +} +``` + +`build_one` runs the same download/md5/build/installpkg steps as sbo-batch-test, +but inside a container via `docker run` instead of `chroot`. The container gets: +the SlackBuild dir mounted read-only, a work dir for output, and (for deps) the +cached .txz mounted in. `sbopkglint` runs host-side on the copied-out package. + +```bash +# build_one <slackbuild-dir> <is_target 0|1> [container-name] +# Runs the build in a throwaway container. Sets ST_STATUS/ST_REASON/ST_TIME. +# Successful builds copy their package to a host workdir; deps are cached and +# the target is linted. Returns 0 on SUCCESS/CACHED, 1 otherwise. +build_one() { + local dir="$1" is_target="${2:-0}" + local prog cat key + prog="$(basename "$dir")"; cat="$(category_of "$dir")"; key="$cat/$prog" + local logf="$RUN_DIR/${cat}_${prog}.log" + local start; start=$(date +%s) + local version; version="$(version_of "$dir")" + + [[ "${HAS_README[$dir]:-}" == "1" ]] && ST_README["$key"]=1 + + # Dep with a version-matching cached package: installpkg it into the shared + # dep-package dir; no build. The target never takes this path. + if [[ "$is_target" != "1" ]]; then + local cached; cached="$(cache_path "$cat" "$prog" "$version")" + if [[ -n "$cached" ]]; then + cp -a "$cached" "$DEPS_DIR/" + { + echo "===== test-build: $prog (from cache) =====" + echo "cached package: $(basename "$cached")" + } >> "$logf" + ST_TIME["$key"]=$(( $(date +%s) - start )) + ST_STATUS["$key"]="CACHED" + return 0 + fi + fi + + # Build in a container. Mounts: + # $dir -> /sbo/pkg (ro, the SlackBuild) + # $DEPS_DIR -> /sbo/deps (rw, already-built dep .txz to installpkg first) + # $BUILD_OUT -> /sbo/out (rw, where the built package is copied out) + # The in-container script installs any deps present, then builds the target, + # writes a status token to /sbo/out/$prog.status, and copies the package out. + local statf="$BUILD_OUT/$prog.status" + rm -f "$statf" + + # fetch deps (removed from -current tree): let the container's sbopkg build + # them first. FETCH_DEPS is the set collected during resolution. + local fetch_list="" + local fp + for fp in "${!FETCH_DEPS[@]}"; do fetch_list+="$fp "; done + + docker run --rm \ + -v "$dir":/sbo/pkg:ro \ + -v "$DEPS_DIR":/sbo/deps \ + -v "$BUILD_OUT":/sbo/out \ + -e PROG="$prog" \ + -e FETCH_LIST="$fetch_list" \ + "$ACTIVE_IMAGE" /bin/bash -s >>"$logf" 2>&1 <<'CONTAINER_EOF' +set -uo pipefail +prog="$PROG" +statf="/sbo/out/$prog.status" + +# 0. install already-built dependency packages (order guaranteed by the host). +for d in /sbo/deps/*.t?z; do + [[ -e "$d" ]] || continue + installpkg --terse "$d" || { echo "INSTALL-FAILED (dep $d)"; echo INSTALL-FAILED > "$statf"; exit 1; } +done + +# 0b. fetch-from-SBo deps via sbopkg (removed from the -current tree). +for f in $FETCH_LIST; do + echo "sbopkg-building fetch dep: $f" + sbopkg -B -i "$f" || { echo "BUILD-FAILED (fetch dep $f)"; echo BUILD-FAILED > "$statf"; exit 1; } +done + +# copy the SlackBuild out of the read-only mount so it can write there. +cp -a /sbo/pkg /sbo/build +cd /sbo/build || { echo BUILD-FAILED > "$statf"; exit 1; } +. ./"$prog".info + +export OUTPUT=/sbo/out +mkdir -p "$OUTPUT" + +echo "===== test-build: $prog =====" +echo "PRGNAM=${PRGNAM:-$prog} VERSION=${VERSION:-?} BUILD=${BUILD:-?} TAG=${TAG:-?}" +echo "uname -m: $(uname -m) OUTPUT=$OUTPUT" +echo "REQUIRES=${REQUIRES:-}" +echo "=================================" + +if [ "$(uname -m)" = "x86_64" ] && [ -n "${DOWNLOAD_x86_64:-}" ] && [ "${DOWNLOAD_x86_64}" != "UNSUPPORTED" ] && [ "${DOWNLOAD_x86_64}" != "UNTESTED" ]; then + DL="$DOWNLOAD_x86_64"; MD="$MD5SUM_x86_64" +else + DL="$DOWNLOAD"; MD="$MD5SUM" +fi + +for u in $DL; do + wget -c --tries=3 "$u" || { echo DOWNLOAD-FAILED > "$statf"; exit 1; } +done + +set -- $MD +for u in $DL; do + f="$(basename "$u")" + want="$1"; shift + got="$(md5sum "$f" | cut -d' ' -f1)" + if [ "$got" != "$want" ]; then + echo "MD5 mismatch on $f: want $want got $got" + echo MD5-MISMATCH > "$statf"; exit 1 + fi +done + +chmod +x ./"$prog".SlackBuild +if ! ./"$prog".SlackBuild; then + echo BUILD-FAILED > "$statf"; exit 1 +fi + +pkg="$(ls -t "$OUTPUT"/"$prog"-*.t?z 2>/dev/null | head -n1)" +if [ -z "$pkg" ]; then + echo "No package produced in $OUTPUT" + echo BUILD-FAILED > "$statf"; exit 1 +fi +if ! installpkg --terse "$pkg"; then + echo INSTALL-FAILED > "$statf"; exit 1 +fi +echo "===== installed files: $(basename "$pkg") =====" +pkgname="$(basename "$pkg")"; pkgname="${pkgname%.t?z}" +cat "/var/log/packages/$pkgname" 2>/dev/null || echo "(package db entry not found)" +echo "=================================" +echo SUCCESS > "$statf" +CONTAINER_EOF + + local status="BUILD-FAILED" + [[ -f "$statf" ]] && status="$(cat "$statf")" + ST_TIME["$key"]=$(( $(date +%s) - start )) + ST_STATUS["$key"]="$status" + + if [[ "$status" == "SUCCESS" ]]; then + # locate the built package copied to the host workdir + local built newest="" + for built in "$BUILD_OUT/${prog}"-*.t?z; do + [[ -e "$built" ]] || continue + [[ -z "$newest" || "$built" -nt "$newest" ]] && newest="$built" + done + if [[ -n "$newest" ]]; then + if [[ "$is_target" != "1" ]]; then + cache_store "$cat" "$prog" "$newest" + # make the dep available to later builds in this run + cp -a "$newest" "$DEPS_DIR/" + else + lint_pkg "$newest" "$logf" + fi + fi + return 0 + fi + ST_REASON["$key"]="see $(basename "$logf")" + return 1 +} +``` + +- [ ] **Step 2: Sanity-check bash syntax** + +Run: `bash -n .extras/test-build` +Expected: no output (syntax OK). + +- [ ] **Step 3: Run the self-check still passes (no regression)** + +Run: `bash .extras/test-logic.sh` +Expected: PASS (build_one is not exercised, but sourcing must not break). + +- [ ] **Step 4: Commit** + +```bash +git add .extras/test-build +SBOLINT=no git commit -m 'test-build: container build step (build_one via docker)' +``` + +--- + +## Task 8: run_target orchestration + summary + +**Files:** +- Modify: `.extras/test-build` + +- [ ] **Step 1: Implement run_target + print_summary + confirm** + +Add to `.extras/test-build` above `main()`. `print_summary` is ported verbatim +(status set is the same). `run_target` replaces the overlay lifecycle with the +per-run docker workdirs and adds the printed-order + Y/n confirm. + +```bash +# confirm_order: print the resolved order (overrides marked) and ask to proceed. +# --yes skips the prompt but the order is still printed. --dry-run never reaches +# here. Returns 0 to proceed, 1 to abort. +confirm_order() { + local target_dir="$1" + echo " build order (${VERSION_ID}):" + local d + for d in "${RESOLVED_ORDER[@]}"; do + local it=0; [[ "$d" == "$target_dir" ]] && it=1 + local rdm=""; [[ "${HAS_README[$d]:-}" == 1 ]] && rdm=" [%README%]" + printf " %-30s %s%s\n" "$(pkg_key "$d")" "$(cache_label "$d" "$it")" "$rdm" + echo "$(pkg_key "$d")" >> "$RUN_DIR/build-order.txt" + done + # note fetch deps (built via sbopkg in-container, not in the order list) + local fp + for fp in "${!FETCH_DEPS[@]}"; do + printf " %-30s %s\n" "$fp" "fetch (sbopkg in container)" + done + [[ $ASSUME_YES -eq 1 ]] && return 0 + local reply + read -rp " Proceed? [Y/n] " reply + [[ -z "$reply" || "$reply" =~ ^[Yy]$ ]] +} + +# run_target <target-slackbuild-dir> +run_target() { + local target_dir="$1" + local tkey; tkey="$(pkg_key "$target_dir")" + + echo + echo "=== Target: $tkey (${VERSION_ID}) ===" + resolve_target "$target_dir" + apply_overrides_to_order + + # Hard resolution failures: report and stop, do not build. + if [[ ${#CYCLES[@]} -gt 0 || ${#UNMET[@]} -gt 0 ]]; then + local why="" + if [[ ${#UNMET[@]} -gt 0 ]]; then + local u + for u in "${!UNMET[@]}"; do why+="unmet:$u(${UNMET[$u]}) "; done + fi + [[ ${#CYCLES[@]} -gt 0 ]] && why+="${CYCLES[*]}" + ST_STATUS["$tkey"]="UNMET-DEP" + ST_REASON["$tkey"]="$why" + echo " resolution failed: $why" + echo " add an override rule ($TB_OVERRIDES) and rerun, or fix the tree." >&2 + return + fi + + if [[ $DRY_RUN -eq 1 ]]; then + echo " build order (dry-run):" + local d + for d in "${RESOLVED_ORDER[@]}"; do + local it=0; [[ "$d" == "$target_dir" ]] && it=1 + local rdm=""; [[ "${HAS_README[$d]:-}" == 1 ]] && rdm=" [%README%]" + printf " %-30s %s%s\n" "$(pkg_key "$d")" "$(cache_label "$d" "$it")" "$rdm" + echo "$(pkg_key "$d")" >> "$RUN_DIR/build-order.txt" + done + local fp + for fp in "${!FETCH_DEPS[@]}"; do + printf " %-30s %s\n" "$fp" "fetch (sbopkg in container)" + done + return + fi + + if ! confirm_order "$target_dir"; then + echo " aborted." + ST_STATUS["$tkey"]="ABORTED" + return + fi + + # Per-run docker workdirs (host side, discarded after the run). + DEPS_DIR="$RUN_DIR/deps"; BUILD_OUT="$RUN_DIR/out" + mkdir -p "$DEPS_DIR" "$BUILD_OUT" + + local d failed_progs=() + for d in "${RESOLVED_ORDER[@]}"; do + local key; key="$(pkg_key "$d")" + local prog; prog="$(basename "$d")" + + if depends_on_failed "$d" failed_progs; then + ST_STATUS["$key"]="BLOCKED-BY-DEP" + ST_REASON["$key"]="blocked by failed dep" + [[ "${HAS_README[$d]:-}" == "1" ]] && ST_README["$key"]=1 + echo " $key: BLOCKED-BY-DEP" + failed_progs+=("$prog") + continue + fi + + local it=0; [[ "$d" == "$target_dir" ]] && it=1 + echo " building $key ..." + if build_one "$d" "$it"; then + echo " $key: ${ST_STATUS[$key]} (${ST_TIME[$key]}s)" + else + echo " $key: ${ST_STATUS[$key]} (${ST_TIME[$key]}s)" + failed_progs+=("$prog") + fi + done +} + +# ============================================================================= +# SUMMARY (ported) +# ============================================================================= +print_summary() { + local total=$SECONDS + local succ=0 fail=0 blocked=0 cached=0 + local summary="$RUN_DIR/summary.log" + + { + echo "test-build run summary" + echo "target: $TARGET_ARG version: $VERSION_ID" + echo + } > "$summary" + + echo + echo "================ SUMMARY ================" + local key + for key in "${!ST_STATUS[@]}"; do + local st="${ST_STATUS[$key]}" rsn="${ST_REASON[$key]:-}" t="${ST_TIME[$key]:-0}" + local rd=""; [[ "${ST_README[$key]:-}" == "1" ]] && rd=" [%README%]" + local col="$C_YEL" + case "$st" in + SUCCESS) col="$C_GRN"; ((succ++)) ;; + CACHED) col="$C_GRN"; ((cached++)) ;; + BLOCKED-BY-DEP|UNMET-DEP|ABORTED) col="$C_YEL"; ((blocked++)) ;; + *) col="$C_RED"; ((fail++)) ;; + esac + printf "%s%-30s %-16s%s %s%s (%ss)\n" "$col" "$key" "$st" "$C_RST" "$rsn" "$rd" "$t" + printf "%-30s %-16s %s%s (%ss)\n" "$key" "$st" "$rsn" "$rd" "$t" >> "$summary" + done + echo "----------------------------------------" + printf "%s%d succeeded%s, %s%d failed%s, %s%d blocked%s, %s%d cached%s, total %ss\n" \ + "$C_GRN" "$succ" "$C_RST" "$C_RED" "$fail" "$C_RST" "$C_YEL" "$blocked" "$C_RST" \ + "$C_GRN" "$cached" "$C_RST" "$total" + echo "logs: $RUN_DIR" + if [[ $fail -eq 0 && $blocked -eq 0 ]]; then + echo "${C_GRN}All green.${C_RST} Safe to build the SBo submission tarball on the host." + fi + { + echo + echo "$succ succeeded, $fail failed, $blocked blocked, $cached cached, total ${total}s" + echo "logs: $RUN_DIR" + } >> "$summary" +} +``` + +Add the two workdir globals near the other globals (top of the flags block): + +```bash +DEPS_DIR="" # per-run host dir of built dep .txz, mounted into the container +BUILD_OUT="" # per-run host dir where built packages are copied out +``` + +- [ ] **Step 2: Sanity-check syntax** + +Run: `bash -n .extras/test-build` +Expected: no output. + +- [ ] **Step 3: Self-check no regression** + +Run: `bash .extras/test-logic.sh` +Expected: PASS. + +- [ ] **Step 4: Commit** + +```bash +git add .extras/test-build +SBOLINT=no git commit -m 'test-build: run_target orchestration + summary + confirm' +``` + +--- + +## Task 9: Wire main(), image presence check, target lookup + +**Files:** +- Modify: `.extras/test-build` + +- [ ] **Step 1: Implement the full main()** + +Replace the current `main()` in `.extras/test-build` with: + +```bash +# Verify the selected image exists locally. It is built by a separate job; this +# script only consumes it. (Later: a local-registry docker pull slots in here.) +require_image() { + if ! docker image inspect "$ACTIVE_IMAGE" >/dev/null 2>&1; then + cat >&2 <<EOF +Image not found locally: $ACTIVE_IMAGE +It is produced by the separate image-builder job (full Slackware $VERSION_ID + +sbo-maintainer-tools + sbopkg), stored on the NAS / local registry. Load or +pull it, then rerun. This script does not build images. +EOF + exit 1 + fi +} + +main() { + parse_args "$@" + init_color + require_config + SBO_TREE_ROOTS=("$ACTIVE_TREE") + load_overrides + + # image + cache namespace (skip the image check on a pure dry-run so the order + # can be inspected without the image present). + if [[ $DRY_RUN -eq 0 ]]; then + command -v docker >/dev/null 2>&1 || { echo "docker not found in PATH." >&2; exit 1; } + require_image + fi + resolve_cache_root + + RUN_DIR="$LOG_ROOT/$(date +%Y-%m-%d_%H-%M-%S)" + mkdir -p "$RUN_DIR" + : > "$RUN_DIR/build-order.txt" + + # Single-package mode: resolve the name in the active tree. + local tdir + if ! tdir="$(find_slackbuild_dir "$TARGET_ARG")"; then + echo "Program not found in the SBo tree ($ACTIVE_TREE): $TARGET_ARG" >&2 + exit 1 + fi + + run_target "$tdir" + print_summary +} +``` + +- [ ] **Step 2: Sanity-check syntax** + +Run: `bash -n .extras/test-build` +Expected: no output. + +- [ ] **Step 3: Dry-run smoke test against the real tree (no docker needed)** + +This needs a real config. If `~/.config/sbo-testbuild/config` is not set up, +create a throwaway one pointing at an existing local SBo tree, then: + +Run: `.extras/test-build --dry-run <some-pkg-in-that-tree>` +Expected: prints `=== Target: ... ===`, a build order, exits 0, no docker call. + +- [ ] **Step 4: Self-check no regression** + +Run: `bash .extras/test-logic.sh` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add .extras/test-build +SBOLINT=no git commit -m 'test-build: wire main, image-presence check, target lookup' +``` + +--- + +## Task 10: Real end-to-end build (user-run) + README + +**Files:** +- Modify: `.extras/test-build` (only if the real run surfaces a bug) +- Create: `.extras/README.test-build` (short usage note) + +This task needs a real image and the buildsystem. Per repo rules the throwaway +docker build is Claude-OK, but the IMAGE must exist first (separate job, not yet +built). So this task is: document usage, and record the end-to-end run as a +manual verification step the user performs once an image is available. + +- [ ] **Step 1: Write the usage README** + +Create `.extras/README.test-build`: + +```markdown +# test-build + +Verify an already-published SBo package still builds on a target Slackware +version inside a throwaway docker container. + +## Setup + + mkdir -p ~/.config/sbo-testbuild + cp .extras/test-build-config.example ~/.config/sbo-testbuild/config + cp .extras/overrides.example ~/.config/sbo-testbuild/overrides + $EDITOR ~/.config/sbo-testbuild/config + +Set the two SBo tree paths, the two image tags, LOG_ROOT, and PKG_CACHE. + +The images (`sbo-testbuild:current` / `:15.0`) are FULL Slackware installs plus +`sbo-maintainer-tools` and `sbopkg`, built by a separate job (not this script). +The script errors if the tagged image is not present locally. + +## Usage + + .extras/test-build <pkg> # build on -current (default) + .extras/test-build --stable <pkg> # build on 15.0 + .extras/test-build --dry-run <pkg> # show the build order, no build + .extras/test-build --yes <pkg> # skip the confirm (order still shown) + .extras/test-build --no-cache <pkg> # rebuild all deps this run + +A green run means it is safe to build the SBo submission tarball on the host. + +## current-vs-stable overrides + +`~/.config/sbo-testbuild/overrides` (see `overrides.example`) encodes deltas +that only apply when building on -current: + + drop: <prog> already in the -current base + rename: <old> -> <new> renamed on -current + fetch: <prog> removed from the -current tree; built via sbopkg + +Start light; add rules only when a real package needs them. + +## Self-check + + bash .extras/test-logic.sh +``` + +- [ ] **Step 2: Commit the docs** + +```bash +git add .extras/README.test-build +SBOLINT=no git commit -m 'test-build: usage README' +``` + +- [ ] **Step 3: Manual end-to-end verification (user, once an image exists)** + +Documented for the user to run when a `sbo-testbuild:*` image is available: + +```bash +# pick a small already-published package with a dep, e.g. one already in the repo +.extras/test-build --yes <pkg> +``` + +Expected: build order printed, deps built (or CACHED on a second run), target +built, `sbopkglint` result shown, summary green, `All green.` line printed. +A second run of the same package should show its deps as `CACHED`. + +Record the outcome. If a real bug surfaces (mount path, sbopkg invocation, +package glob), fix it in `.extras/test-build` and re-run; add a self-check case +if the bug was in pure logic. + +--- + +## Self-Review + +Run this checklist against the spec after the plan is written: + +**1. Spec coverage:** +- purpose / one-target-per-run: Task 2 (`parse_args`), Task 9 (`main`) - covered +- two versions, current default: Task 2 (`--stable`, `select_version_paths`) - covered +- resolve local tree + topo: Task 3 - covered +- overrides drop/rename/fetch + UNMET stop: Task 4, stop in Task 8 `run_target` - covered +- external config + overrides file: Task 2, Task 4 - covered +- external image, presence check, no bake: Task 9 `require_image` - covered +- container build (download/md5/build/installpkg): Task 7 - covered +- fetch via sbopkg: Task 7 (`FETCH_LIST` loop) - covered +- dep cache, digest-namespaced, --no-cache, target always fresh: Task 5, wired Task 7/8 - covered +- target lint: Task 7 `lint_pkg`, called in Task 7 build_one - covered +- confirm order + --yes still prints: Task 8 `confirm_order` - covered +- --dry-run: Task 8 `run_target`, Task 9 skips image check - covered +- report + logs (per-pkg, summary, build-order): Task 8 `print_summary`, Task 7 logs - covered +- status values incl CACHED: Task 8 `print_summary` - covered +- self-check parity: Tasks 1-6 - covered + +**2. Placeholder scan:** no TBD/TODO in code steps; every code step shows full +code. The one deferred item (real end-to-end run) is explicit and gated on an +external image, not a placeholder. + +**3. Type consistency:** function + global names match across tasks: +`RESOLVED_ORDER`, `FETCH_DEPS`, `OV_DROP/OV_RENAME/OV_FETCH`, `CACHE_ROOT`, +`ACTIVE_TREE/ACTIVE_IMAGE`, `DEPS_DIR/BUILD_OUT`, `cache_decision/cache_path/ +cache_store/cache_label`, `select_version_paths`, `apply_overrides_to_order`, +`apply_rename`, `load_overrides`, `resolve_cache_root`, `require_image`, +`build_one(dir,is_target)`, `run_target`, `confirm_order`, `print_summary`. +Consistent. + +## Notes on deliberate simplifications + +- Single-package mode only (no category-folder / all / queue modes). This repo + bumps one package at a time. Extension point is `main`'s target lookup. +- `-j` parallelism dropped entirely (was a no-op stub upstream). Builds serial. +- fetch deps are not topo-ordered among themselves; sbopkg resolves their own + deps. Start light per the spec; revisit if a fetch dep needs another fetch dep. +``` + diff --git a/docs/plans/2026-07-13-image-builder.md b/docs/plans/2026-07-13-image-builder.md new file mode 100644 index 0000000..0c61051 --- /dev/null +++ b/docs/plans/2026-07-13-image-builder.md @@ -0,0 +1,790 @@ +# sbo-testbuild image builder — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Build the three chained bash scripts that produce `sbo-testbuild:current` / `sbo-testbuild:15.0` docker images from NAS Slackware trees and push them to a LAN registry, consumed by `.extras/test-build`. + +**Architecture:** Adapt the forge `slackware/docker-images` scripts (`bootstrap.sh`, `build-full-image.sh`) into `.extras/image-builder/`. Add a third `build-sbo-testbuild.sh` layer. Share config + helpers via `config` and `lib.sh`. Pure-logic helpers are unit-tested with an assert-based self-check; the docker chain is verified manually on the VM. + +**Tech Stack:** bash, docker, slackpkg/installpkg (Slackware), `registry:2`. + +**Reference:** design spec at `.extras/docs/specs/2026-07-13-image-builder-design.md`. Forge originals cloned at (scratchpad) `docker-images/scripts/` — re-clone with `git clone slackware_forge:slackware/docker-images.git` if gone. + +--- + +## File Structure + +``` +.extras/image-builder/ +├── config REGISTRY, MIRROR, VARIANTS, PKGDIR (sourced by all) +├── lib.sh _log/_warn/_err, fetch(), txz_hash(), require_mount() +├── bootstrap.sh base image FROM scratch (root; installpkg from NAS) +├── build-full-image.sh full image FROM base (slackpkg all series) +├── build-sbo-testbuild.sh testbuild image FROM full (installpkg sbopkg + tools) +├── test-image-builder.sh assert-based self-check for lib.sh logic +└── README VM-side setup checklist +``` + +Responsibilities: +- **config** — the only place hostnames/paths/variants live. No script hardcodes them. +- **lib.sh** — all shared, testable logic. Sourced by the three build scripts and by the test. No side effects at source time (only function defs + nothing else). +- **bootstrap/full/testbuild** — one image each, thin: guard, gate, generate Dockerfile, build, push. +- **test-image-builder.sh** — sources lib.sh, exercises fetch/txz_hash/require_mount with asserts. No docker. + +--- + +## Task 1: config + lib.sh skeleton + +**Files:** +- Create: `.extras/image-builder/config` +- Create: `.extras/image-builder/lib.sh` + +- [ ] **Step 1: Write config** + +Create `.extras/image-builder/config`: + +```sh +# Shared configuration for the sbo-testbuild image builder. +# Sourced by bootstrap.sh, build-full-image.sh, build-sbo-testbuild.sh, tests. + +REGISTRY="docker.noland.dnx:5000" +MIRROR="file:///mnt/nas" +VARIANTS=(current 15.0) # x86_64 only for now +PKGDIR="/opt/sbo-testbuild/pkgs" # sbopkg + sbo-maintainer-tools .txz +HASH_DIR="/var/cache/sbo-testbuild" # ChangeLog hashes for rebuild gating +``` + +- [ ] **Step 2: Write lib.sh with logging + require_mount** + +Create `.extras/image-builder/lib.sh`: + +```sh +# Shared helpers for the image builder. Source, do not execute. +# Sourcing must have no side effects beyond defining functions. + +_log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] [${LOG_TAG:-image-builder}] $*"; } +_warn() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] [${LOG_TAG:-image-builder}] WARNING: $*" >&2; } +_err() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] [${LOG_TAG:-image-builder}] ERROR: $*" >&2; exit 1; } + +# mirror_path MIRROR RELPATH +# Join a MIRROR (file:///... or http(s)://...) with a relative path. +mirror_path() { + local mirror="$1" rel="$2" + printf '%s/%s' "${mirror%/}" "${rel#/}" +} + +# is_local_mirror MIRROR -> 0 if file:// or bare absolute path, else 1 +is_local_mirror() { + case "$1" in + file://*) return 0 ;; + http://*|https://*) return 1 ;; + /*) return 0 ;; + *) return 1 ;; + esac +} + +# strip_scheme URL -> filesystem path for a file:// or bare-local URL +strip_scheme() { + local u="$1" + case "$u" in + file://*) printf '%s' "${u#file://}" ;; + *) printf '%s' "$u" ;; + esac +} + +# fetch SRC DEST +# Copy SRC (a full mirror URL) to DEST. Local mirror -> cp, http -> curl. +# Returns non-zero on failure; caller decides whether that is fatal. +fetch() { + local src="$1" dest="$2" + if is_local_mirror "$src"; then + local path; path="$(strip_scheme "$src")" + [[ -f "$path" ]] || return 1 + cp "$path" "$dest" + else + curl -sSfL --connect-timeout 60 "$src" -o "$dest" + fi +} + +# txz_hash DIR +# Stable hash of the set of .txz files in DIR (names + contents), order +# independent. Empty/absent dir -> empty string. +txz_hash() { + local dir="$1" + [[ -d "$dir" ]] || { printf ''; return 0; } + local files; files=$(find "$dir" -maxdepth 1 -name '*.txz' -type f | sort) + [[ -n "$files" ]] || { printf ''; return 0; } + # Hash sorted "name sha256" lines so reordering does not change output. + while IFS= read -r f; do + printf '%s %s\n' "$(basename "$f")" "$(sha256sum "$f" | cut -d' ' -f1)" + done <<< "$files" | sha256sum | cut -d' ' -f1 +} + +# require_mount VERSION +# Assert the NAS tree for VERSION is mounted (file:// mirror only). +# x86_64 tree dir is slackware64-${VERSION}. +require_mount() { + local version="$1" + is_local_mirror "$MIRROR" || return 0 # http mirror: nothing to check + local base; base="$(strip_scheme "$MIRROR")" + local dir="${base}/slackware64-${version}" + [[ -d "$dir" ]] || _err "NAS tree not mounted: ${dir}" +} +``` + +- [ ] **Step 3: Commit** + +```bash +git add .extras/image-builder/config .extras/image-builder/lib.sh +git commit -m 'image-builder: config + lib.sh helpers' +``` + +--- + +## Task 2: lib.sh self-check + +**Files:** +- Create: `.extras/image-builder/test-image-builder.sh` + +- [ ] **Step 1: Write the test** + +Create `.extras/image-builder/test-image-builder.sh`: + +```sh +#!/bin/bash +# Assert-based self-check for lib.sh pure logic. No docker, no network. +# Run: bash .extras/image-builder/test-image-builder.sh +set -u +cd "$(dirname "$0")" +source ./lib.sh + +pass=0 fail=0 +check() { # DESC EXPECTED ACTUAL + if [[ "$2" == "$3" ]]; then pass=$((pass+1)); + else fail=$((fail+1)); echo "FAIL: $1"; echo " expected: [$2]"; echo " actual: [$3]"; fi +} +check_rc() { # DESC EXPECTED_RC ACTUAL_RC + if [[ "$2" == "$3" ]]; then pass=$((pass+1)); + else fail=$((fail+1)); echo "FAIL: $1 (rc expected $2 got $3)"; fi +} + +tmp="$(mktemp -d)" +trap 'rm -rf "$tmp"' EXIT + +# --- mirror_path --- +check "mirror_path trailing slash" \ + "file:///mnt/nas/slackware64-current/PACKAGES.TXT" \ + "$(mirror_path 'file:///mnt/nas/' 'slackware64-current/PACKAGES.TXT')" +check "mirror_path no trailing slash" \ + "file:///mnt/nas/x/y" "$(mirror_path 'file:///mnt/nas' '/x/y')" + +# --- is_local_mirror --- +is_local_mirror "file:///mnt/nas"; check_rc "is_local file://" 0 $? +is_local_mirror "/mnt/nas"; check_rc "is_local bare abs" 0 $? +is_local_mirror "https://x/y"; check_rc "is_local https" 1 $? + +# --- strip_scheme --- +check "strip_scheme file://" "/mnt/nas" "$(strip_scheme 'file:///mnt/nas')" +check "strip_scheme bare" "/mnt/nas" "$(strip_scheme '/mnt/nas')" + +# --- fetch: local file resolves --- +echo hello > "$tmp/src.txt" +fetch "file://$tmp/src.txt" "$tmp/dst.txt"; check_rc "fetch local ok" 0 $? +check "fetch local content" "hello" "$(cat "$tmp/dst.txt" 2>/dev/null)" + +# --- fetch: missing local file errors --- +fetch "file://$tmp/nope.txt" "$tmp/x.txt"; check_rc "fetch local missing" 1 $? + +# --- txz_hash: stable + content-sensitive --- +mkdir -p "$tmp/pkgs" +printf a > "$tmp/pkgs/sbopkg-1.txz" +printf b > "$tmp/pkgs/tools-1.txz" +h1="$(txz_hash "$tmp/pkgs")" +h2="$(txz_hash "$tmp/pkgs")" +check "txz_hash stable" "$h1" "$h2" +printf c > "$tmp/pkgs/tools-1.txz" # change content +h3="$(txz_hash "$tmp/pkgs")" +check_rc "txz_hash changes on content" 0 "$([[ "$h1" != "$h3" ]] && echo 0 || echo 1)" +check "txz_hash empty dir" "" "$(txz_hash "$tmp/empty-nope")" + +# --- require_mount: missing mount errors (subshell to catch _err exit) --- +MIRROR="file://$tmp/no-such-root" +( require_mount current ) 2>/dev/null; check_rc "require_mount missing" 1 $? +mkdir -p "$tmp/mnt/slackware64-current" +MIRROR="file://$tmp/mnt" +( require_mount current ) 2>/dev/null; check_rc "require_mount present" 0 $? +MIRROR="https://example/x" +( require_mount current ) 2>/dev/null; check_rc "require_mount http skips" 0 $? + +echo "----" +echo "PASS: $pass FAIL: $fail" +[[ "$fail" -eq 0 ]] +``` + +- [ ] **Step 2: Run — expect PASS** + +Run: `bash .extras/image-builder/test-image-builder.sh` +Expected: final line `PASS: N FAIL: 0`, exit 0. + +If any FAIL, fix `lib.sh` (the test encodes the intended behavior). + +- [ ] **Step 3: Commit** + +```bash +git add .extras/image-builder/test-image-builder.sh +git commit -m 'image-builder: lib.sh self-check' +``` + +--- + +## Task 3: bootstrap.sh (base image) + +**Files:** +- Create: `.extras/image-builder/bootstrap.sh` +- Reference: forge `docker-images/scripts/bootstrap.sh` + +This is the forge `bootstrap.sh` adapted: source `config`+`lib.sh`, drop i586, +`file://` fetch via `fetch()`, LAN registry, `sbo-base` name, mount guard. + +- [ ] **Step 1: Write bootstrap.sh** + +Create `.extras/image-builder/bootstrap.sh`. Start from the forge original and apply exactly these changes; everything else (the `download_pkgtxt`, `find_package`, rootfs configuration, cleanup, Dockerfile-from-scratch blocks) is copied verbatim from the forge script: + +1. Header block: + +```sh +#!/bin/bash +# bootstrap.sh — build the sbo-base:{ver} image FROM scratch from NAS trees. +# Run as root (installpkg). Adapted from forge slackware/docker-images. +set -euo pipefail +HERE="$(cd "$(dirname "$0")" && pwd)" +source "${HERE}/config" +LOG_TAG=bootstrap +source "${HERE}/lib.sh" + +REGISTRY_IMAGE="${REGISTRY}/sbo-base" +[[ "${EUID}" -eq 0 ]] || _err "run as root." +mkdir -p "${HASH_DIR}" +``` + +2. Argument parsing: keep `--force` and `--version` only; **remove `--arch`** + (x86_64 fixed). Variant list comes from config: + +```sh +FORCE=false +OPT_VERSION="" +while [[ $# -gt 0 ]]; do + case "$1" in + --version) OPT_VERSION="$2"; shift 2 ;; + --force) FORCE=true; shift ;; + *) _err "unknown argument: $1" ;; + esac +done +if [[ -n "${OPT_VERSION}" ]]; then + BUILD_VARIANTS=("${OPT_VERSION}") +else + BUILD_VARIANTS=("${VARIANTS[@]}") +fi +``` + +3. Copy the forge `PACKAGES=( ... )` array verbatim (lines 80–187 of the forge + original — the full base package list). + +4. Replace `changelog_changed`: read ChangeLog via `fetch` from the NAS, + x86_64-only repo key: + +```sh +# changelog_changed VERSION -> 0 if changed (or --force), else 1 +changelog_changed() { + local version="$1" + local repo_key="slackware64-${version}" + local hash_file="${HASH_DIR}/${repo_key}.sha256" + local url; url="$(mirror_path "${MIRROR}" "${repo_key}/ChangeLog.txt")" + local tmp; tmp="$(mktemp)" + if ! fetch "${url}" "${tmp}"; then + rm -f "${tmp}" + _warn "${repo_key}: cannot read ChangeLog (${url}); skipping." + return 1 + fi + local live; live=$(sha256sum "${tmp}" | cut -d' ' -f1); rm -f "${tmp}" + if [[ "${FORCE}" == "true" ]]; then + echo "${live}" > "${hash_file}"; return 0 + fi + local stored=""; [[ -f "${hash_file}" ]] && stored=$(cat "${hash_file}") + if [[ "${live}" == "${stored}" ]]; then + _log "${repo_key}: ChangeLog unchanged; skipping."; return 1 + fi + _log "${repo_key}: ChangeLog changed."; echo "${live}" > "${hash_file}"; return 0 +} +``` + +5. Copy `download_pkgtxt` and `find_package` verbatim from the forge original, + but change the two `curl -sfL ... "${URL}" -o "${DEST}"` download lines in + `download_pkgtxt` to `fetch "${URL}" "${DEST}"` (drops the http assumption). + +6. `build_variant`: take `VERSION` only (no ARCH). Set: + +```sh +build_variant() { + local VERSION="$1" + require_mount "${VERSION}" + local REPO_KEY="slackware64-${VERSION}" + local PKG_PATH; PKG_PATH="$(mirror_path "${MIRROR}" "${REPO_KEY}")" + local TAG="${REGISTRY_IMAGE}:${VERSION}" + local SLACKPKG_MIRROR="https://slackware.nl/slackware/slackware64-${VERSION}/" + ... +} +``` + + The `SLACKPKG_MIRROR` written into the image stays an https slackware.nl URL + (it is used later by slackpkg inside the container, which has internet; the + NAS `file://` mount only feeds the host-side build). Keep the rest of the + forge `build_variant` body verbatim, except: + - the package **download loop** uses `fetch "${URL}" "${PKGCACHE}/${FILENAME}"` + instead of `curl`. + - the final `docker build ... && docker push` targets `${TAG}` as above. + +7. Main loop over `BUILD_VARIANTS`, one arg: + +```sh +for VERSION in "${BUILD_VARIANTS[@]}"; do + if changelog_changed "${VERSION}"; then + build_variant "${VERSION}" + fi +done +``` + +- [ ] **Step 2: Syntax check** + +Run: `bash -n .extras/image-builder/bootstrap.sh` +Expected: no output, exit 0. + +- [ ] **Step 3: shellcheck (if available)** + +Run: `shellcheck -x .extras/image-builder/bootstrap.sh || true` +Expected: no errors (warnings about unreachable `_err` exit are fine). If shellcheck absent, skip. + +- [ ] **Step 4: Commit** + +```bash +git add .extras/image-builder/bootstrap.sh +git commit -m 'image-builder: bootstrap.sh base image from NAS' +``` + +--- + +## Task 4: build-full-image.sh + +**Files:** +- Create: `.extras/image-builder/build-full-image.sh` +- Reference: forge `docker-images/scripts/build-full-image.sh` + +Forge `build-full-image.sh` adapted: source config+lib, drop i586, LAN +registry, `sbo-base`→`sbo-full` names. + +- [ ] **Step 1: Write build-full-image.sh** + +Create `.extras/image-builder/build-full-image.sh`. From the forge original, apply exactly: + +1. Header: + +```sh +#!/bin/bash +# build-full-image.sh — sbo-full:{ver} FROM sbo-base:{ver}, all series. +# Adapted from forge slackware/docker-images. No root needed. +set -euo pipefail +HERE="$(cd "$(dirname "$0")" && pwd)" +source "${HERE}/config" +LOG_TAG=full +source "${HERE}/lib.sh" + +BASE_IMAGE="${REGISTRY}/sbo-base" +FULL_IMAGE="${REGISTRY}/sbo-full" +DIGEST_LABEL="sbo.full.base-digest" +``` + +2. Arg parsing: `--force` and `--version` only (no `--arch`), variant list from + config — identical shape to bootstrap Task 3 Step 1.2, but `BUILD_VARIANTS`. + +3. `derive_tags VERSION`: + +```sh +derive_tags() { + local VERSION="$1" + BASE_TAG="${BASE_IMAGE}:${VERSION}" + FULL_TAG="${FULL_IMAGE}:${VERSION}" +} +``` + +4. Keep `needs_rebuild` verbatim from the forge original (it already takes + BASE_TAG/FULL_TAG args and uses `_log`/`_warn`/`FORCE`). + +5. `build_variant VERSION`: drop the ARCH/ARCH_SUFFIX lines; call + `derive_tags "${VERSION}"`. Keep the forge Dockerfile heredoc **verbatim** + (the `slackpkg update` + all-series install + cleanup + labels). Only the + `LABEL org.opencontainers.image.title/description` strings lose the + `(${ARCH})` suffix; leave them or hardcode `(x86_64)`. + +6. Main loop over `BUILD_VARIANTS`: + +```sh +for VERSION in "${BUILD_VARIANTS[@]}"; do + build_variant "${VERSION}" +done +``` + +- [ ] **Step 2: Syntax check** + +Run: `bash -n .extras/image-builder/build-full-image.sh` +Expected: exit 0. + +- [ ] **Step 3: shellcheck (if available)** + +Run: `shellcheck -x .extras/image-builder/build-full-image.sh || true` + +- [ ] **Step 4: Commit** + +```bash +git add .extras/image-builder/build-full-image.sh +git commit -m 'image-builder: build-full-image.sh' +``` + +--- + +## Task 5: build-sbo-testbuild.sh + +**Files:** +- Create: `.extras/image-builder/build-sbo-testbuild.sh` + +New script (no forge equivalent). FROM sbo-full, installpkg the two .txz, +gate on full digest AND .txz-set hash. + +- [ ] **Step 1: Write build-sbo-testbuild.sh** + +Create `.extras/image-builder/build-sbo-testbuild.sh`: + +```sh +#!/bin/bash +# build-sbo-testbuild.sh — sbo-testbuild:{ver} FROM sbo-full:{ver}. +# Adds sbopkg + sbo-maintainer-tools from prebuilt .txz in PKGDIR. +# Rebuilds when the full image OR the .txz set changes. +set -euo pipefail +HERE="$(cd "$(dirname "$0")" && pwd)" +source "${HERE}/config" +LOG_TAG=testbuild +source "${HERE}/lib.sh" + +FULL_IMAGE="${REGISTRY}/sbo-full" +TB_IMAGE="${REGISTRY}/sbo-testbuild" +DIGEST_LABEL="sbo.testbuild.full-digest" +PKGS_LABEL="sbo.testbuild.pkgs-hash" + +FORCE=false +OPT_VERSION="" +while [[ $# -gt 0 ]]; do + case "$1" in + --version) OPT_VERSION="$2"; shift 2 ;; + --force) FORCE=true; shift ;; + *) _err "unknown argument: $1" ;; + esac +done +if [[ -n "${OPT_VERSION}" ]]; then + BUILD_VARIANTS=("${OPT_VERSION}") +else + BUILD_VARIANTS=("${VARIANTS[@]}") +fi + +# Verify the required .txz are present before doing anything. +require_pkgs() { + [[ -d "${PKGDIR}" ]] || _err "PKGDIR missing: ${PKGDIR}" + compgen -G "${PKGDIR}/sbopkg-*.txz" >/dev/null \ + || _err "no sbopkg-*.txz in ${PKGDIR}" + compgen -G "${PKGDIR}/sbo-maintainer-tools-*.txz" >/dev/null \ + || _err "no sbo-maintainer-tools-*.txz in ${PKGDIR}" +} + +# needs_rebuild FULL_TAG TB_TAG PKGS_HASH -> 0 if rebuild needed +needs_rebuild() { + local full_tag="$1" tb_tag="$2" pkgs_hash="$3" + [[ "${FORCE}" == "true" ]] && { _log " --force."; return 0; } + + local full_digest + full_digest=$(docker inspect --format '{{index .RepoDigests 0}}' \ + "${full_tag}" 2>/dev/null || echo "") + [[ -n "${full_digest}" ]] || { _warn " no full digest; rebuilding."; return 0; } + + local rec_digest rec_pkgs + rec_digest=$(docker inspect \ + --format "{{index .Config.Labels \"${DIGEST_LABEL}\"}}" \ + "${tb_tag}" 2>/dev/null || echo "") + rec_pkgs=$(docker inspect \ + --format "{{index .Config.Labels \"${PKGS_LABEL}\"}}" \ + "${tb_tag}" 2>/dev/null || echo "") + + if [[ "${full_digest}" == "${rec_digest}" && "${pkgs_hash}" == "${rec_pkgs}" ]]; then + _log " full image and .txz set unchanged; skipping." + return 1 + fi + _log " full image or .txz set changed; rebuilding." + return 0 +} + +build_variant() { + local VERSION="$1" + local FULL_TAG="${FULL_IMAGE}:${VERSION}" + local TB_TAG="${TB_IMAGE}:${VERSION}" + _log "=== ${TB_TAG} ===" + + _log " Pulling ${FULL_TAG}..." + docker pull "${FULL_TAG}" + + local PKGS_HASH; PKGS_HASH="$(txz_hash "${PKGDIR}")" + + if ! needs_rebuild "${FULL_TAG}" "${TB_TAG}" "${PKGS_HASH}"; then + return 0 + fi + + local FULL_DIGEST + FULL_DIGEST=$(docker inspect --format '{{index .RepoDigests 0}}' "${FULL_TAG}") + + local WORKDIR; WORKDIR="$(mktemp -d /tmp/sbo-testbuild.XXXXXX)" + trap "rm -rf '${WORKDIR}'" RETURN + mkdir -p "${WORKDIR}/pkgs" + cp "${PKGDIR}"/sbopkg-*.txz "${PKGDIR}"/sbo-maintainer-tools-*.txz "${WORKDIR}/pkgs/" + + cat > "${WORKDIR}/Dockerfile" <<DOCKERFILE +FROM ${FULL_TAG} +LABEL maintainer="danix <danix@danix.xyz>" +COPY pkgs/*.txz /tmp/pkgs/ +RUN installpkg /tmp/pkgs/*.txz && rm -rf /tmp/pkgs +RUN /sbin/ldconfig +LABEL org.opencontainers.image.created="$(date -u +%Y-%m-%dT%H:%M:%SZ)" +LABEL org.opencontainers.image.title="SBo test-build env ${VERSION} (x86_64)" +LABEL org.opencontainers.image.description="Slackware ${VERSION} full + sbopkg + sbo-maintainer-tools" +LABEL slackware.version="${VERSION}" +LABEL ${DIGEST_LABEL}="${FULL_DIGEST}" +LABEL ${PKGS_LABEL}="${PKGS_HASH}" +CMD ["/bin/bash"] +DOCKERFILE + + local BUILD_FLAGS=() + [[ "${FORCE}" == "true" ]] && BUILD_FLAGS+=(--no-cache) + _log " Building ${TB_TAG}..." + docker build "${BUILD_FLAGS[@]}" -t "${TB_TAG}" "${WORKDIR}" + _log " Pushing ${TB_TAG}..." + docker push "${TB_TAG}" + _log "=== Done: ${TB_TAG} ===" +} + +require_pkgs +for VERSION in "${BUILD_VARIANTS[@]}"; do + build_variant "${VERSION}" +done +``` + +- [ ] **Step 2: Syntax check** + +Run: `bash -n .extras/image-builder/build-sbo-testbuild.sh` +Expected: exit 0. + +- [ ] **Step 3: shellcheck (if available)** + +Run: `shellcheck -x .extras/image-builder/build-sbo-testbuild.sh || true` + +- [ ] **Step 4: Commit** + +```bash +git add .extras/image-builder/build-sbo-testbuild.sh +git commit -m 'image-builder: build-sbo-testbuild.sh' +``` + +--- + +## Task 6: per-variant isolation + +The spec requires one variant failing not to block the other, non-zero exit at +the end. Add this to all three scripts' main loops. + +**Files:** +- Modify: `.extras/image-builder/bootstrap.sh` (main loop) +- Modify: `.extras/image-builder/build-full-image.sh` (main loop) +- Modify: `.extras/image-builder/build-sbo-testbuild.sh` (main loop) + +- [ ] **Step 1: Wrap each main loop** + +In each script, replace the plain `for VERSION ...` main loop with an +isolation wrapper. For **bootstrap.sh** (which also has the changelog gate): + +```sh +rc=0 +for VERSION in "${BUILD_VARIANTS[@]}"; do + if changelog_changed "${VERSION}"; then + if ! ( build_variant "${VERSION}" ); then + _warn "variant ${VERSION} failed; continuing." + rc=1 + fi + fi +done +exit "${rc}" +``` + +For **build-full-image.sh** and **build-sbo-testbuild.sh** (no changelog gate): + +```sh +rc=0 +for VERSION in "${BUILD_VARIANTS[@]}"; do + if ! ( build_variant "${VERSION}" ); then + _warn "variant ${VERSION} failed; continuing." + rc=1 + fi +done +exit "${rc}" +``` + +Note: `build_variant` runs in a subshell `( )` so a `set -e` abort inside it +(or an `_err` exit) is contained and the loop proceeds to the next variant. +For build-sbo-testbuild.sh keep the `require_pkgs` call **before** the loop +(a missing .txz is fatal for all variants, not per-variant). + +- [ ] **Step 2: Syntax check all three** + +Run: `for f in bootstrap build-full-image build-sbo-testbuild; do bash -n .extras/image-builder/$f.sh; done` +Expected: exit 0, no output. + +- [ ] **Step 3: Re-run lib self-check (unaffected, sanity)** + +Run: `bash .extras/image-builder/test-image-builder.sh` +Expected: `FAIL: 0`. + +- [ ] **Step 4: Commit** + +```bash +git add .extras/image-builder/bootstrap.sh .extras/image-builder/build-full-image.sh .extras/image-builder/build-sbo-testbuild.sh +git commit -m 'image-builder: per-variant isolation, non-zero exit on failure' +``` + +--- + +## Task 7: README + +**Files:** +- Create: `.extras/image-builder/README` + +- [ ] **Step 1: Write README** + +Create `.extras/image-builder/README` with the VM-side setup, verbatim: + +``` +sbo-testbuild image builder +=========================== + +Builds the docker images that .extras/test-build consumes: + docker.noland.dnx:5000/sbo-testbuild:current + docker.noland.dnx:5000/sbo-testbuild:15.0 + +Three scripts, chained (see .extras/docs/specs/2026-07-13-image-builder-design.md): + bootstrap.sh sbo-base:{ver} FROM scratch, base pkgs from NAS + build-full-image.sh sbo-full:{ver} FROM base, all series + build-sbo-testbuild.sh sbo-testbuild:{ver} FROM full, + sbopkg + tools + +All settings live in ./config. + +VM setup (docker.noland.dnx, Slackware x86_64, 4 vCPU / 4 GB / 80 GB) +-------------------------------------------------------------------- +1. Install docker; enable the daemon. + +2. NFS-mount the two NAS trees read-only, named to match: + /mnt/nas/slackware64-current -> -current mirror tree + /mnt/nas/slackware64-15.0 -> 15.0 mirror tree + Each is a full mirror (PACKAGES.TXT, ChangeLog.txt, slackware64/, patches/, + extra/). Root must be able to read them (bootstrap runs installpkg as root). + +3. Run a LAN registry: + docker run -d --restart=always -p 5000:5000 \ + -v /opt/registry/data:/var/lib/registry --name registry registry:2 + +4. Mark the registry insecure (plain HTTP) on the VM AND every pulling client + (this dev box, the buildsystem VM). In /etc/docker/daemon.json: + { "insecure-registries": ["docker.noland.dnx:5000"] } + then restart docker. + +5. Drop the two prebuilt packages (built once, re-drop on upstream bumps): + /opt/sbo-testbuild/pkgs/sbopkg-*.txz + /opt/sbo-testbuild/pkgs/sbo-maintainer-tools-*.txz + +6. Install the nightly cron (root): + 0 4 * * * /path/to/.extras/image-builder/bootstrap.sh >> /var/log/sbo-testbuild.log 2>&1 + 15 4 * * * /path/to/.extras/image-builder/build-full-image.sh >> /var/log/sbo-testbuild.log 2>&1 + 30 4 * * * /path/to/.extras/image-builder/build-sbo-testbuild.sh >> /var/log/sbo-testbuild.log 2>&1 + +7. Ensure docker.noland.dnx resolves on the LAN (static IP or DNS). + +Manual first run +---------------- + ./bootstrap.sh --version current --force + ./build-full-image.sh --version current --force + ./build-sbo-testbuild.sh --version current --force +Then confirm: + docker pull docker.noland.dnx:5000/sbo-testbuild:current + docker run --rm docker.noland.dnx:5000/sbo-testbuild:current sbopkg -V + +Flags: --force (rebuild unconditionally), --version <current|15.0> (one variant). + +Tests +----- + bash test-image-builder.sh # pure-logic self-check, no docker +``` + +- [ ] **Step 2: Commit** + +```bash +git add .extras/image-builder/README +git commit -m 'image-builder: README with VM setup checklist' +``` + +--- + +## Task 8: executable bits + final check + +**Files:** +- Modify (mode): the four `.sh` scripts + +- [ ] **Step 1: chmod +x the scripts** + +```bash +chmod +x .extras/image-builder/bootstrap.sh \ + .extras/image-builder/build-full-image.sh \ + .extras/image-builder/build-sbo-testbuild.sh \ + .extras/image-builder/test-image-builder.sh +``` + +- [ ] **Step 2: Run the self-check once more** + +Run: `bash .extras/image-builder/test-image-builder.sh` +Expected: `PASS: N FAIL: 0`, exit 0. + +- [ ] **Step 3: Syntax-check all four** + +Run: `for f in .extras/image-builder/*.sh; do bash -n "$f" || echo "SYNTAX FAIL: $f"; done` +Expected: no `SYNTAX FAIL` lines. + +- [ ] **Step 4: Commit** + +```bash +git add .extras/image-builder +git commit -m 'image-builder: mark scripts executable' +``` + +--- + +## Deferred (not in this plan) + +- **Real end-to-end chain** on the VM (bootstrap→full→testbuild producing a + pushed `sbo-testbuild:current`): the user runs it once the VM/registry/mounts + exist, and reports back. Mirrors `.extras/test-build` Task 10. +- **`installed_in_base`** in `.extras/test-build`: now unblockable (query the + built image's package db) but tracked separately in the project memory + `test-build-image-builder-todo`. +- **i586** variants. +- **Registry auth/TLS** (LAN plain HTTP for now). +``` diff --git a/docs/specs/2026-07-13-docker-test-build-design.md b/docs/specs/2026-07-13-docker-test-build-design.md new file mode 100644 index 0000000..9f14d1e --- /dev/null +++ b/docs/specs/2026-07-13-docker-test-build-design.md @@ -0,0 +1,258 @@ +# Docker test-build script - design + +Date: 2026-07-13 + +## Purpose + +A single bash script that verifies an already-published SBo package still +builds cleanly on a target Slackware version, inside a throwaway docker +container. This is the "test-build" step of this repo's maintenance loop +(track upstream -> bump -> **test-build** -> make tarball). + +The container is disposable, so unlike a real `slackrepo build` this runs +directly (no host install), and Claude may run it (per the repo's +`Who builds SlackBuilds` exception). + +## Scope and non-goals + +- One target package per run: `test-build <pkg>`. +- Two Slackware targets: `-current` (default) and `15.0` (`--stable`). +- Resolves the target's SBo dependency tree from the LOCAL SBo tree, builds + deps then target in the container, reports, discards everything. +- NOT a redistributable-package builder. The container's built `.txz` is + throwaway; the SBo submission tarball is made separately (post-commit hook) + only after a green run. +- Does NOT wrap slackrepo or sbo-batch-test. It REUSES sbo-batch-test's + resolver + build logic, adapted, but is an independent script. +- Shares the host kernel (docker). No kernel-module build claims. + +## Why not the overlay approach (sbo-batch-test) + +`sbo-batch-test` already resolves deps + builds + reports, but against a +LOCAL overlayfs over a 15.0 base. In docker the container IS the disposable +base, and overlayfs-over-docker-storage (itself overlayfs) is fragile. So we +drop the overlay layer and build directly in the container. We port +sbo-batch-test's dep resolver and its `build_one` chroot heredoc +(download -> md5 -> build -> installpkg -> status token) into a `docker run`. + +## Architecture + +Single self-contained bash script (`test-build`, location TBD in +implementation - likely `.extras/`). Host does resolution + orchestration; +the container does the actual building. + +``` +host: parse args, load config +host: pick version -> image tag + SBo tree +host: resolve dep tree (local tree, topo sort) [ported from sbo-batch-test] +host: apply current-vs-stable overrides +host: print final build order, Y/n confirm (--yes: still prints first) + docker run --rm -v <tree>:ro -v <pkg>:ro -v <cache/digest>:ro <image>: + for each dep in order: + cache hit (version match) -> installpkg from cache (CACHED) + else -> download/md5/build/installpkg + cache + target: download/md5/build/installpkg (always fresh) + target: sbopkglint on the built .txz +host: collect per-package results, print color summary +container discarded (--rm) +``` + +### Images (external input, not built by this script) + +The script does NOT build images. It consumes a ready image per version, +tagged locally: + +- current: `sbo-testbuild:current` +- 15.0: `sbo-testbuild:15.0` + +Each image is a FULL, patched Slackware install (SBo tests against a full +install; a minimal base causes false "missing dependency" results, a lesson +from sbo-batch-tester) plus `sbo-maintainer-tools` (for `sbopkglint`) and +`sbopkg` configured to that version's SBo repo (`:current` -> -current repo, +`:15.0` -> 15.0 repo). + +**How the images are produced is out of scope for this script.** A separate +nightly job (a LAN machine, images stored on the NAS, later served from a +local registry) builds and refreshes them. That builder is its own tool, +specced separately. + +For now the script assumes the tagged image is present locally: it runs +`docker run` against the tag and errors with a clear message if the image is +missing (pointing at the image-builder job). When the local registry exists, +a `docker pull` of the tag slots in ahead of the run via the same config, +no other change. + +Per-run cost is just the SlackBuild's own build time + ~1s container start. + +### Config (external file) + +`~/.config/sbo-testbuild/config`, empty defaults in-script, sourced if +present (same pattern as sbo-batch-test). Keys: + +```sh +SBO_TREE_CURRENT=/path/to/SBo-current # local SBo tree, -current +SBO_TREE_STABLE=/path/to/SBo-15.0 # local SBo tree, 15.0 +IMAGE_CURRENT=sbo-testbuild:current # ready image tag (built elsewhere) +IMAGE_STABLE=sbo-testbuild:15.0 +LOG_ROOT=/path/to/logs +PKG_CACHE=/path/to/cache # dep cache; empty = disabled +``` + +The version flag selects BOTH the image and the tree together (default +current; `--stable` / `15.0` switches both), so tree and image never mismatch. + +### current-vs-stable overrides + +A data file next to the config, `~/.config/sbo-testbuild/overrides`. It +encodes the known deltas between building on 15.0 and on -current. Applied +ONLY when the target version is `-current` (15.0 is the SBo baseline). Three +rule kinds: + +``` +# dep present in 15.0 but already in -current base -> drop from order +drop: rust +# dep renamed on -current -> map old -> new before lookup +rename: python3-foo -> foo +# dep removed from the -current tree -> fetch from SBo instead of local tree +fetch: somelib +``` + +Format: one rule per line, `kind: value` (rename uses `old -> new`), `#` +comments and blank lines ignored. + +### Dependency resolution + unknown deps + +Resolver is ported from sbo-batch-test (`resolve_target` / `_resolve_visit`, +DFS topo sort + cycle detection, `installed_in_base` check). Runs on the host +against the selected tree. + +After resolving and applying overrides, every dep must be accounted for: +in base, in the tree, or covered by a `fetch:` rule. If a dep is none of +those (and no rule covers it), the script **stops before building**, reports +it as UNMET, and asks the user to add an override rule and rerun, or abort. +Never silently skip or guess. + +### Confirmation + +Always print the final build order before building, with overrides marked +(dropped / renamed / fetch). Then `Y/n` to proceed. `--yes` skips the prompt +for non-interactive/agent runs but STILL prints the order first. + +`--dry-run` resolves, applies overrides, prints the order, and exits without +building (no confirm). + +### Build in container (ported from build_one) + +Deps then target, in order. For each package, inside the container: + +1. download sources (arch-specific `DOWNLOAD_x86_64`/`MD5SUM_x86_64` when + present, else `DOWNLOAD`/`MD5SUM`) +2. verify md5 +3. `bash <pkg>.SlackBuild` with `OUTPUT` forced to a known dir +4. `installpkg` the resulting `.txz` +5. write a status token read back by the host + +`fetch:` deps (removed from the -current tree) are pulled via the baked-in +`sbopkg`, which points at the image's matching SBo repo; everything else +builds from the mounted local tree. Start light: handle only the few known +`fetch:` cases, add rules as real packages need them. + +### Dependency cache + +Ported from sbo-batch-tester (`cache_decision`/`cache_path`/`cache_store`/ +`version_of`). A host dir caches built dep packages so consecutive runs reuse +them instead of rebuilding the whole tree. + +- Config `PKG_CACHE=/path` enables it; empty (default) disables. `--no-cache` + forces a full rebuild for one run. +- Bind-mounted read-only into the container; a cached dep is `installpkg`ed + instead of built (status `CACHED`). Storing is host-side: a freshly built + package is written to `OUTPUT` (a mounted dir the host reads back), and the + host copies it into the cache after the build (`cache_store`), so the cache + mount stays read-only. The **target always builds fresh** and refreshes its + own cache entry; it is never `CACHED`. +- Key = prog + version (build/arch/tag ignored). Layout mirrors the SBo tree: + `$PKG_CACHE/<image-digest>/<category>/<prog>/<prog>-<ver>-...txz`, one .txz + per prog. +- **Invalidation by image digest.** The cache is namespaced by the running + image's digest. When the digest changes (image updated by the nightly job), + the old namespace is stale: its deps were built against the previous image, + so they are ignored and rebuilt once under the new digest. This also keeps + the `:current` and `:15.0` caches separate (different images, different + digests). Old digest namespaces can be pruned. +- A per-dep version bump (`bump: OLD -> NEW`) rebuilds and re-caches that dep. +- Build-order line shows the outcome per dep: `cached (1.1)`, + `rebuild: 1.0 -> 1.1`, or `build (new)`; `--dry-run` shows the same without + building. + +### Target lint + +After the target builds, run `sbopkglint` on its `.txz` (tool baked into the +image). Fail-soft: findings reported but do not change SUCCESS; missing tool +prints a skip note. Target only (deps are vetted). + +### Report + +Per-package status + a color summary (screen) and a plain-text log under +`LOG_ROOT/<timestamp>/`: + +``` +<timestamp>/ + <prog>.log per-package full build/install output + summary.log plain recap + build-order.txt the resolved+overridden order actually used +``` + +Status values (from sbo-batch-test): +`SUCCESS CACHED DOWNLOAD-FAILED MD5-MISMATCH BUILD-FAILED INSTALL-FAILED +BLOCKED-BY-DEP UNMET-DEP`. `CACHED` = a dep installed from the cache instead +of rebuilt (target is never CACHED). `%README%` deps flagged as a reminder, +not built. + +A fully green run tells the user it is safe to make the SBo submission tarball +on the host (the container is no longer needed). + +## Options (planned) + +| Option | Effect | +|--------|--------| +| `--stable` / `15.0` | Target 15.0 (image + tree). Default is -current. | +| `--dry-run` | Resolve + apply overrides + print order, no build. | +| `--yes` | Skip the Y/n confirm (still prints the order). | +| `--no-cache` | Rebuild all deps this run, ignore/refresh the cache. | +| `--no-color` | Disable ANSI (auto-off when not a TTY). | +| `-h`, `--help` | Usage. | + +## Self-check + +A `test-logic.sh` (host, no docker) covering the pure logic: + +- dep topo order + cycle detection (ported tests from sbo-batch-test) +- override application: drop removes, rename maps, fetch marks source +- unknown-dep -> UNMET stop +- BLOCKED-BY-DEP propagation +- cache decision: `cached`/`rebuild: OLD -> NEW`/`build (new)`, digest + namespacing, version-bump eviction (ported from sbo-batch-tester's tests) + +Docker/build/installpkg paths are out of the self-check's reach (verified by +running a real target), same boundary as sbo-batch-test. + +## Resolved decisions + +- **Image**: a ready FULL-install image per version (full pkg set + + `sbo-maintainer-tools` + `sbopkg` on that version's repo), built by a + separate nightly job, NOT by this script. Script consumes it by tag. +- **Missing image**: error clearly, point at the image-builder job. A local + registry `docker pull` slots in later via the same tag, no script change. +- **`fetch:`**: via the image's `sbopkg` (per-version repo). Start with only + the few known cases; grow the override rules as needed. +- **Script location**: `.extras/` (a tool, not a package). + +## Out of scope / later + +- The nightly image-builder (full-install + tools + sbopkg per version, NAS + storage, local registry). Its own tool, specced when the LAN machine is set + up. +- Local registry pull step in this script (slots into the same config tag). +``` + diff --git a/docs/specs/2026-07-13-image-builder-design.md b/docs/specs/2026-07-13-image-builder-design.md new file mode 100644 index 0000000..9ee1a11 --- /dev/null +++ b/docs/specs/2026-07-13-image-builder-design.md @@ -0,0 +1,216 @@ +# sbo-testbuild image builder — design + +Date: 2026-07-13 +Status: approved (design) + +## Purpose + +Produce and serve the docker images the `.extras/test-build` tool consumes. +`.extras/test-build` verifies a published SBo package still builds on a target +Slackware version inside a throwaway container; it does **not** build the +images, it only checks image presence by tag and errors if absent. This is the +missing image-builder job. + +Images produced (x86_64 only for now): + +- `sbo-testbuild:current` — Slackware -current, full install + sbopkg + + sbo-maintainer-tools +- `sbo-testbuild:15.0` — Slackware 15.0, same + +Served from a LAN docker registry so `.extras/test-build`'s `require_image` +can `docker pull` them, and so `installed_in_base` can later query the image's +package db. + +## Infrastructure + +Self-hosted on a Slackware VM (proxmox node), fully offline from the public +internet for the Slackware bits: + +- **Host**: `docker.noland.dnx` (LAN DNS), Slackware x86_64, 4 vCPU / 4 GB RAM + / 80 GB disk. +- **NAS trees**: two standard Slackware mirror trees NFS-mounted read-only: + - `/mnt/nas/slackware64-current` + - `/mnt/nas/slackware64-15.0` + Each is a full mirror tree (`PACKAGES.TXT`, `ChangeLog.txt`, `slackware64/` + series dirs, `patches/`, `extra/`). +- **Registry**: `registry:2` container on the VM, port 5000, plain HTTP (LAN + only). Pull path `docker.noland.dnx:5000/sbo-testbuild:{tag}`. +- **Local .txz drop**: `/opt/sbo-testbuild/pkgs/` holds `sbopkg-*.txz` and + `sbo-maintainer-tools-*.txz`, provided by the user (sbopkg packaged from the + user's SBo repo). Installed into the image, not built in it (hermetic, no + build-time deps). + +Both `docker.noland.dnx` and every client that pulls (this dev box, the +`buildsystem` VM) need the registry marked insecure in +`/etc/docker/daemon.json`: +`{"insecure-registries":["docker.noland.dnx:5000"]}`, then `docker` restarted. + +i586 is out of scope for now; the variant list is written to extend to it +later without restructuring. + +## Architecture + +Three scripts, chained, adapted from the forge repo +`slackware/docker-images` (`scripts/bootstrap.sh`, `build-full-image.sh`, +`build-builder-image.sh`). The forge `build-builder-image.sh` is **not** used: +the full image already ships the toolchain (series d, kde, x, etc.), so a +separate builder layer is redundant here. + +``` +bootstrap.sh FROM scratch; installpkg base pkgs read from NAS + → sbo-base:{ver} via file://; configure rootfs for container use. + +build-full-image.sh FROM sbo-base:{ver}; slackpkg install every series + → sbo-full:{ver} (a ap d e f k kde l n t tcl x xap xfce y). + +build-sbo-testbuild.sh FROM sbo-full:{ver}; installpkg sbopkg + + → sbo-testbuild:{ver} sbo-maintainer-tools from /opt/sbo-testbuild/pkgs. +``` + +All three push to `docker.noland.dnx:5000`. + +### Namespacing + +Forge scripts push to `registry.slackware.nl/slackware/slackware*`. Re-namespace +to the LAN registry with our own image names: + +- `docker.noland.dnx:5000/sbo-base:{ver}` +- `docker.noland.dnx:5000/sbo-full:{ver}` +- `docker.noland.dnx:5000/sbo-testbuild:{ver}` + +`sbo-base` and `sbo-full` are intermediate; `.extras/test-build` only pulls +`sbo-testbuild`. + +## Configuration + +Single shared file `config`, sourced by all scripts and the test: + +```sh +REGISTRY=docker.noland.dnx:5000 +MIRROR=file:///mnt/nas +VARIANTS=(current 15.0) # x86_64 only for now +PKGDIR=/opt/sbo-testbuild/pkgs # sbopkg + sbo-maintainer-tools .txz +``` + +The forge scripts hardcode `MIRROR=https://slackware.nl/slackware` and +`REGISTRY=registry.slackware.nl/...`; both move to the shared config. + +### NAS path mapping + +bootstrap derives the per-variant repo dir as `slackware64-${VERSION}` for +x86_64 (`DIRSUFFIX=64`). With `MIRROR=file:///mnt/nas` this resolves to +`file:///mnt/nas/slackware64-current/...` and `.../slackware64-15.0/...`, +matching the mount points. Mounts must be named to match. + +## Patches to the forge scripts + +1. **`file://`-aware fetch.** Forge downloads `PACKAGES.TXT` and packages with + `curl` over http. Replace with one helper in `lib.sh` that detects a + `file://` (or bare-local) MIRROR and uses `cp`, otherwise `curl`. curl's + own `file://` support is unreliable across builds, so branch explicitly. +2. **Drop i586.** `ALL_VARIANTS` becomes the config `VARIANTS` (x86_64 only). + All the `ARCH_SUFFIX`/`-i586` machinery collapses to the x86_64 path. +3. **Re-namespace + LAN registry** (see Namespacing). +4. **MIRROR → `file:///mnt/nas`** (see config). + +## Data flow and rebuild gating + +Nightly cron on the VM (root; `bootstrap` needs `installpkg`): + +```cron + 0 4 * * * bootstrap.sh +15 4 * * * build-full-image.sh +30 4 * * * build-sbo-testbuild.sh +``` + +Rebuild gating (kept from the forge scripts, extended): + +- **bootstrap**: sha256 of the NAS `ChangeLog.txt` (read via `file://`) + compared to a stored hash under `HASH_DIR`; skip the variant if unchanged. +- **build-full-image**: base image digest recorded as a label on the full + image; pull base, compare, skip if unchanged. +- **build-sbo-testbuild**: full image digest label, same pattern; **plus** a + hash of the `.txz` set in `PKGDIR`, so a sbopkg / sbo-maintainer-tools bump + forces a rebuild even when the full image is unchanged. + +Net effect: a quiet NAS tree makes the whole chain no-op in seconds; a +-current churn cascades base → full → testbuild. + +### Local-registry digest consistency + +The forge scripts read the base/full digest via `docker inspect RepoDigests` +after `docker pull`. Self-hosted, those intermediate images live in the LAN +registry, so pull-then-inspect must target `docker.noland.dnx:5000`. Since +every image name is derived from `REGISTRY`, re-namespacing handles this with +no extra logic. + +## Error handling + +- `set -euo pipefail` in every script (forge already has it). +- **Missing NAS mount**: guard `[[ -d /mnt/nas/slackware64-$ver ]]` first; + `_err` if absent. +- **Missing `.txz`**: `build-sbo-testbuild` errors before building if + `PKGDIR` lacks a `sbopkg-*.txz` or `sbo-maintainer-tools-*.txz` — never push + a testbuild image missing sbopkg. +- **Registry unreachable**: `docker push` fails under `set -e`; the chain + aborts. The base/full may already be built locally; cron retries next night. + Acceptable. +- **Per-variant isolation**: one variant failing (e.g. current) must not block + the other (15.0). Each variant runs wrapped; log the failure, continue, and + exit non-zero at the end so cron logs surface it. + +## Testing + +Pure-logic pieces get a `test-image-builder.sh` self-check, matching the +existing `.extras/test-build` pattern (`.extras/test-logic.sh`, assert-based, +no framework). No docker daemon in tests — hermetic. + +- **`file://` fetch helper**: local file resolves via cp; missing local file + errors; an http MIRROR still routes to curl. +- **`.txz`-set hash**: stable across file reordering; changes when a file's + content changes. +- **variant guard**: a missing mount dir yields non-zero. + +The real end-to-end chain is verified by a manual first run on the VM (the user +runs it and reports back), the same division of labor as `.extras/test-build` +Task 10. + +## Layout + +``` +.extras/image-builder/ +├── config REGISTRY, MIRROR, VARIANTS, PKGDIR +├── lib.sh shared: _log/_warn/_err, fetch, txz_hash, guards +├── bootstrap.sh base image (root; installpkg from NAS) +├── build-full-image.sh full image (slackpkg all series) +├── build-sbo-testbuild.sh testbuild image (installpkg sbopkg + tools) +├── test-image-builder.sh pure-logic self-check +└── README VM-side setup, reproducible +``` + +Lives in `.extras/` for now (per the user), relocatable later. + +The README carries the user-side checklist so it is reproducible and not +trapped in chat: + +- VM specs (4/4/80), Slackware x86_64, docker installed. +- Two NFS mounts (ro), root-readable, named `slackware64-current` / + `slackware64-15.0` under `/mnt/nas`. +- `registry:2` container: `-v /opt/registry/data:/var/lib/registry`, + `--restart=always`, port 5000. +- `/etc/docker/daemon.json` insecure-registry entry on the VM **and** on every + pulling client (this dev box, `buildsystem`), docker restarted. +- Drop `sbopkg-*.txz` and `sbo-maintainer-tools-*.txz` into + `/opt/sbo-testbuild/pkgs/`. +- Install the three cron lines. +- VM static IP / LAN DNS for `docker.noland.dnx`. + +## Out of scope + +- i586 images (extensible later via `VARIANTS` / arch machinery). +- Building sbopkg / sbo-maintainer-tools from source in the image (installed + from prebuilt `.txz`). +- The forge `build-builder-image.sh` toolchain layer (redundant with full). +- `installed_in_base` implementation in `.extras/test-build` — unblocked by + this work (queries the built image's package db) but tracked separately. +- Registry auth / TLS (LAN-only plain HTTP). diff --git a/image-builder/README b/image-builder/README new file mode 100644 index 0000000..8711b1d --- /dev/null +++ b/image-builder/README @@ -0,0 +1,69 @@ +sbo-testbuild image builder +=========================== + +Builds the docker images that test-build consumes: + docker.noland.dnx:5000/sbo-testbuild:current + docker.noland.dnx:5000/sbo-testbuild:15.0 + +Three scripts, chained (see docs/specs/2026-07-13-image-builder-design.md): + bootstrap.sh sbo-base:{ver} FROM scratch, base pkgs from NAS + build-full-image.sh sbo-full:{ver} FROM base, all series + build-sbo-testbuild.sh sbo-testbuild:{ver} FROM full, + sbopkg + tools + +All settings live in ./config. + +VM setup (docker.noland.dnx, Slackware x86_64, 4 vCPU / 4 GB / 80 GB) +-------------------------------------------------------------------- +1. Install docker; enable the daemon. + +2. NFS-mount the two NAS trees read-only, named to match: + /mnt/nas/slackware64-current -> -current mirror tree + /mnt/nas/slackware64-15.0 -> 15.0 mirror tree + Each is a full mirror (PACKAGES.TXT, ChangeLog.txt, slackware64/, patches/, + extra/). Root must be able to read them (bootstrap runs installpkg as root). + +3. Run a LAN registry: + docker run -d --restart=always -p 5000:5000 \ + -v /opt/registry/data:/var/lib/registry --name registry registry:2 + +4. Mark the registry insecure (plain HTTP) on the VM AND every pulling client + (this dev box, the buildsystem VM). In /etc/docker/daemon.json: + { "insecure-registries": ["docker.noland.dnx:5000"] } + then restart docker. + +5. Drop the two prebuilt packages (built once, re-drop on upstream bumps): + /opt/sbo-testbuild/pkgs/sbopkg-*.txz + /opt/sbo-testbuild/pkgs/sbo-maintainer-tools-*.txz + +6. Install the nightly cron (root). The NAS repos sync at 01:00 and 02:00, so + the chain runs after and both variants are ready well before the ~09:00 work + start. No --force: each script self-gates (bootstrap on the ChangeLog hash, + full-image on the base-image digest, build-sbo-testbuild on the full-image + digest + tools .txz hash), so an unchanged night is a cheap no-op. -current + moves daily and rebuilds most nights; 15.0 is frozen stable and rebuilds only + on a real repo update. Deployed schedule on docker.noland.dnx: + # -current (ready ~04:35) + 0 3 * * * /path/to/sbo-dockerbuild/image-builder/bootstrap.sh --version current >> /var/log/sbo-testbuild.log 2>&1 + 20 3 * * * /path/to/sbo-dockerbuild/image-builder/build-full-image.sh --version current >> /var/log/sbo-testbuild.log 2>&1 + 30 4 * * * /path/to/sbo-dockerbuild/image-builder/build-sbo-testbuild.sh --version current >> /var/log/sbo-testbuild.log 2>&1 + # 15.0 (ready ~06:35) + 0 5 * * * /path/to/sbo-dockerbuild/image-builder/bootstrap.sh --version 15.0 >> /var/log/sbo-testbuild.log 2>&1 + 20 5 * * * /path/to/sbo-dockerbuild/image-builder/build-full-image.sh --version 15.0 >> /var/log/sbo-testbuild.log 2>&1 + 30 6 * * * /path/to/sbo-dockerbuild/image-builder/build-sbo-testbuild.sh --version 15.0 >> /var/log/sbo-testbuild.log 2>&1 + +7. Ensure docker.noland.dnx resolves on the LAN (static IP or DNS). + +Manual first run +---------------- + ./bootstrap.sh --version current --force + ./build-full-image.sh --version current --force + ./build-sbo-testbuild.sh --version current --force +Then confirm: + docker pull docker.noland.dnx:5000/sbo-testbuild:current + docker run --rm docker.noland.dnx:5000/sbo-testbuild:current sbopkg -V + +Flags: --force (rebuild unconditionally), --version <current|15.0> (one variant). + +Tests +----- + bash test-image-builder.sh # pure-logic self-check, no docker diff --git a/image-builder/bootstrap.sh b/image-builder/bootstrap.sh new file mode 100755 index 0000000..273afc8 --- /dev/null +++ b/image-builder/bootstrap.sh @@ -0,0 +1,564 @@ +#!/bin/bash +# +# Copyright (C) 2026 Danilo M. <danix@danix.xyz> +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# bootstrap.sh — build the sbo-base:{ver} image FROM scratch from NAS trees. +# Run as root (installpkg). Adapted from forge slackware/docker-images. +set -euo pipefail +HERE="$(cd "$(dirname "$0")" && pwd)" +source "${HERE}/config" +LOG_TAG=bootstrap +source "${HERE}/lib.sh" + +REGISTRY_IMAGE="${REGISTRY}/sbo-base" +[[ "${EUID}" -eq 0 ]] || _err "run as root." +mkdir -p "${HASH_DIR}" + +# ============================================================================ +# Package list +# +# Names only - the version and build suffix are resolved automatically +# from PACKAGES.TXT for the x86_64 target architecture. +# +# slackpkg and its runtime dependencies (perl, gnupg2, gpgme) are included +# so users can run slackpkg inside the container without extra setup. +# ============================================================================ + +PACKAGES=( + # Base system + aaa_base + coreutils + aaa_glibc-solibs + aaa_libraries + aaa_terminfo + pam + cracklib + libpwquality + acl + attr + bash + bin + brotli + bzip2 + c-ares + cpio + curl + cyrus-sasl + devs + dialog + diffutils + duktape + elvis + etc + file + findutils + flex + gawk + gnutls + grep + groff + gzip + iproute2 + iptables + jansson + less + libcgroup + libpsl + librsvg + libtermcap + mtr + ncurses + net-tools + network-scripts + nghttp2 + nghttp3 + ngtcp2 + nvi + openssh + patch + pcre2 + pinentry + pkgtools + polkit + procps-ng + quota + rsync + screen + sed + elogind + shadow + sharutils + strace + sudo + sysfsutils + sysvinit + sysvinit-scripts + tar + time + tree + eudev + usbutils + utempter + util-linux + wget + which + whois + xz + zlib + # Package management - series ap + slackpkg + # Perl runtime required by slackpkg - series d + perl + # e2fsprogs provides libcom_err.so.2 required by perl on 15.0 + e2fsprogs + # libunistring - Unicode string library required by gnupg2 dependencies + libunistring + # gnupg2 runtime libraries (series l) - required for gpg to start at all + libgpg-error + libgcrypt + libassuan + libksba + npth + sqlite + icu4c + # GPG/PKCS support for slackpkg (series l) + gpgme + # GnuPG binaries (series n) + # gnupg = version 1, called by slackpkg on Slackware 15.0 + # gnupg2 = version 2, called by slackpkg on -current + gnupg + gnupg2 + # Full OpenSSL and CA bundle - required for HTTPS inside the container + openssl + ca-certificates +) + +# ============================================================================ +# Argument parsing +# ============================================================================ + +FORCE=false +OPT_VERSION="" +while [[ $# -gt 0 ]]; do + case "$1" in + --version) OPT_VERSION="$2"; shift 2 ;; + --force) FORCE=true; shift ;; + *) _err "unknown argument: $1" ;; + esac +done +if [[ -n "${OPT_VERSION}" ]]; then + BUILD_VARIANTS=("${OPT_VERSION}") +else + BUILD_VARIANTS=("${VARIANTS[@]}") +fi + +# ============================================================================ +# ChangeLog tracking +# ============================================================================ + +# changelog_changed VERSION -> 0 if changed (or --force), else 1 +changelog_changed() { + local version="$1" + require_mount "${version}" # loud fail if NAS unmounted, not silent skip + local repo_key="slackware64-${version}" + local hash_file="${HASH_DIR}/${repo_key}.sha256" + local url; url="$(mirror_path "${MIRROR}" "${repo_key}/ChangeLog.txt")" + local tmp; tmp="$(mktemp)" + if ! fetch "${url}" "${tmp}"; then + rm -f "${tmp}" + _warn "${repo_key}: cannot read ChangeLog (${url}); skipping." + return 1 + fi + local live; live=$(sha256sum "${tmp}" | cut -d' ' -f1); rm -f "${tmp}" + if [[ "${FORCE}" == "true" ]]; then + echo "${live}" > "${hash_file}"; return 0 + fi + local stored=""; [[ -f "${hash_file}" ]] && stored=$(cat "${hash_file}") + if [[ "${live}" == "${stored}" ]]; then + _log "${repo_key}: ChangeLog unchanged; skipping."; return 1 + fi + _log "${repo_key}: ChangeLog changed."; echo "${live}" > "${hash_file}"; return 0 +} + +# ============================================================================ +# Package discovery via PACKAGES.TXT +# +# Search order: patches/ -> main repo -> extra/ +# This mirrors Slackware's own install priority: a patched package in +# patches/ always takes precedence over the same package in the main tree. +# ============================================================================ + +# download_pkgtxt PKGTXT_DIR PKG_PATH +# Downloads PACKAGES.TXT from patches/, the repo root, and extra/ into +# PKGTXT_DIR. Sets global associative array PKGTXT[loc] -> local file path. +# Uses -L to follow HTTP redirects (some mirrors use them) and checks that +# the downloaded file is non-empty before marking it as available. +declare -gA PKGTXT=() + +download_pkgtxt() { + local PKGTXT_DIR="$1" + local PKG_PATH="$2" + mkdir -p "${PKGTXT_DIR}" + PKGTXT=() + local LOC URL DEST + for LOC in patches main extra; do + case "${LOC}" in + patches) URL="${PKG_PATH}/patches/PACKAGES.TXT" ;; + main) URL="${PKG_PATH}/PACKAGES.TXT" ;; + extra) URL="${PKG_PATH}/extra/PACKAGES.TXT" ;; + esac + DEST="${PKGTXT_DIR}/${LOC}.txt" + if fetch "${URL}" "${DEST}" && [[ -s "${DEST}" ]]; then + PKGTXT["${LOC}"]="${DEST}" + _log " PACKAGES.TXT [${LOC}]: $(wc -l < "${DEST}") lines" + else + rm -f "${DEST}" + _log " PACKAGES.TXT [${LOC}]: not available at ${URL}" + fi + done +} + +# find_package NAME +# Searches PKGTXT files in order: patches -> main -> extra. +# Prints the path relative to PKG_PATH, e.g.: +# slackware64/a/bash-5.2-x86_64-1.txz +# +# Base name extraction uses the Slackware convention: the filename has the +# form PKGBASE-VERSION-ARCH-BUILD.txz where VERSION, ARCH and BUILD are +# guaranteed to contain no dashes. Reversing, cutting the last three fields, +# and reversing again always yields the correct PKGBASE: +# echo aaa_glibc-solibs-2.42-x86_64-1.txz | rev | cut -d- -f4- | rev +# -> aaa_glibc-solibs +find_package() { + local PKG="$1" + [[ "${#PKGTXT[@]}" -gt 0 ]] || { + _warn "find_package called with empty PKGTXT array - PACKAGES.TXT not loaded" + return 1 + } + for LOC in patches main extra; do + [[ -v "PKGTXT[${LOC}]" ]] || continue + local RESULT + RESULT=$(awk -v want="${PKG}" ' + /^PACKAGE NAME:/ { + filename = $NF + # PKGBASE = all fields except the last three (VERSION ARCH BUILD) + n = split(filename, parts, "-") + base = parts[1] + for (i = 2; i <= n-3; i++) base = base "-" parts[i] + if (base == want) found = filename + } + /^PACKAGE LOCATION:/ && found { + loc = $NF; sub(/^\.\//, "", loc) + print loc "/" found + found = ""; exit + } + ' "${PKGTXT[${LOC}]}") + if [[ -n "${RESULT}" ]]; then + echo "${RESULT}" + return 0 + fi + done + return 1 +} + +# ============================================================================ +# Build one variant +# ============================================================================ + +build_variant() { + local VERSION="$1" + require_mount "${VERSION}" + local REPO_KEY="slackware64-${VERSION}" + local PKG_PATH; PKG_PATH="$(mirror_path "${MIRROR}" "${REPO_KEY}")" + local TAG="${REGISTRY_IMAGE}:${VERSION}" + # Bake the LAN HTTP mirror into the base image: the full-image build serves + # the NFS mirror over HTTP on the bridge (see build-full-image.sh), rooted + # at the variant's mirror tree, so slackpkg pulls from LAN not the internet. + # The URL is variant-agnostic (the server's --directory selects the tree). + local SLACKPKG_MIRROR="http://${HTTP_MIRROR_HOST}:${HTTP_MIRROR_PORT}/" + + _log "=== Building ${TAG} from ${PKG_PATH} ===" + + local WORKDIR + WORKDIR="$(mktemp -d /tmp/slackware-bootstrap.XXXXXX)" + # shellcheck disable=SC2064 + trap "rm -rf '${WORKDIR}'" EXIT + + local ROOTFS="${WORKDIR}/rootfs" + local PKGCACHE="${WORKDIR}/packages" + local PKGTXT_DIR="${WORKDIR}/pkgtxt" + mkdir -p "${ROOTFS}" "${PKGCACHE}" + + # --- Download PACKAGES.TXT index files --- + download_pkgtxt "${PKGTXT_DIR}" "${PKG_PATH}" + + # --- Download packages --- + _log "Locating and downloading packages..." + local PKG RELPATH FILENAME URL + for PKG in "${PACKAGES[@]}"; do + RELPATH=$(find_package "${PKG}") || { + _warn " '${PKG}' not found in any PACKAGES.TXT - skipping" + continue + } + FILENAME="${RELPATH##*/}" + URL="${PKG_PATH}/${RELPATH}" + _log " ${FILENAME}" + fetch "${URL}" "${PKGCACHE}/${FILENAME}" || { + _warn " Failed to download ${URL}" + } + done + + # --- Install packages --- + _log "Installing packages..." + for PKG in "${PKGCACHE}"/*.txz; do + [[ -f "${PKG}" ]] || continue + ROOT="${ROOTFS}" installpkg --terse "${PKG}" + done + + # --- Run ldconfig to create library symlinks --- + # Some packages (notably aaa_libraries on stable releases) install + # versioned library files (e.g. libtinfo.so.6.3) and rely on ldconfig + # to create the unversioned symlinks (e.g. libtinfo.so.6). On -current + # the symlinks are included in the package tarballs directly; on 15.0 + # they are not. Running ldconfig here ensures the symlinks exist before + # any chroot operations and before the dependency check below. + _log "Running ldconfig..." + chroot "${ROOTFS}" /sbin/ldconfig + + # --- Check for missing library dependencies --- + # Run ldd on key binaries inside the chroot and report anything missing + # before it surfaces as a runtime failure inside the container. + _log "Checking for missing library dependencies in key binaries..." + local DEPCHECK_OK=true + for BIN in /usr/bin/gpg /usr/bin/gpg2 /usr/sbin/slackpkg /usr/bin/perl /bin/wget /usr/bin/openssl; do + [[ -f "${ROOTFS}${BIN}" ]] || continue + local MISSING_LIBS + MISSING_LIBS=$(chroot "${ROOTFS}" ldd "${BIN}" 2>/dev/null | grep "not found" || true) + if [[ -n "${MISSING_LIBS}" ]]; then + _warn "${BIN} has unresolved dependencies:" + while IFS= read -r lib; do + _warn " ${lib}" + done <<< "${MISSING_LIBS}" + DEPCHECK_OK=false + fi + done + if [[ "${DEPCHECK_OK}" == "true" ]]; then + _log "Dependency check passed." + else + _warn "Add the missing packages to PACKAGES and rebuild with --force." + fi + + # --- Prepare rootfs for container use --- + _log "Configuring rootfs for container use..." + + # Remove the empty update.d hook directory so that update-ca-certificates + # does not look for run-parts (which ships with dcron, not installed here). + rm -rf "${ROOTFS}/etc/ca-certificates/update.d" + + # Disable init scripts that manage hardware absent in a container + local SVC + for SVC in rc.acpid rc.pcmcia rc.setterm rc.udev; do + [[ -f "${ROOTFS}/etc/rc.d/${SVC}" ]] && chmod -x "${ROOTFS}/etc/rc.d/${SVC}" + done + + # Each container generates its own SSH host keys on first start + rm -f "${ROOTFS}"/etc/ssh/*key* + + # /etc/mtab should reflect the container's actual mounts + rm -f "${ROOTFS}/etc/mtab" + ln -s /proc/mounts "${ROOTFS}/etc/mtab" + + # No hardware clock in a container + sed -i -e '/^if \[ -x \/sbin\/hwclock/,/^fi$/s/^/#/' \ + "${ROOTFS}/etc/rc.d/rc.S" 2>/dev/null || true + + # The filesystem write-check would drop the container into a recovery shell + sed -i -e '/^if touch \/fsrwtestfile/,/^fi$/s/^/#/' \ + "${ROOTFS}/etc/rc.d/rc.S" 2>/dev/null || true + + # Skip all filesystem checks at boot + touch "${ROOTFS}/etc/fastboot" + + # setterm is not useful in a container + sed -i -e '/\/bin\/setterm/s/^/# /' \ + "${ROOTFS}/etc/rc.d/rc.M" 2>/dev/null || true + + # Cannot write to the hardware clock at shutdown + sed -i -e '/systohc/s/^/# /' \ + "${ROOTFS}/etc/rc.d/rc.6" 2>/dev/null || true + + # Container-appropriate fstab + cat > "${ROOTFS}/etc/fstab" <<'FSTAB' +devtmpfs /dev devtmpfs defaults 0 0 +devpts /dev/pts devpts gid=5,mode=620 0 0 +tmpfs /dev/shm tmpfs defaults,nodev,nosuid,mode=1777 0 0 +FSTAB + + # No TTYs in a container; disable agetty and reduce console definitions + sed -i -e '/agetty/s/^c/#c/' \ + "${ROOTFS}/etc/inittab" 2>/dev/null || true + sed -i -e '/^c3\|^c4\|^c5\|^c6/s/^/# /' \ + "${ROOTFS}/etc/inittab" 2>/dev/null || true + + # Invalidate the root password (containers use other means of access) + sed -i -e '/^root/s/^root::/root:!:/' \ + "${ROOTFS}/etc/shadow" 2>/dev/null || true + + # Terminal configuration + printf 'export TERM=linux\n' > "${ROOTFS}/etc/profile.d/term.sh" + chmod +x "${ROOTFS}/etc/profile.d/term.sh" + + # Source /etc/profile from .bashrc so interactive sessions are fully + # initialised (Docker sets / as the home directory for root in containers) + printf '. /etc/profile\n' > "${ROOTFS}/.bashrc" + + # DNS resolver + printf 'nameserver 1.1.1.1\nnameserver 8.8.4.4\n' \ + >> "${ROOTFS}/etc/resolv.conf" + + # Configure slackpkg for silent non-interactive use in a container + if [[ -f "${ROOTFS}/etc/slackpkg/slackpkg.conf" ]]; then + sed -i 's/DIALOG=on/DIALOG=off/' "${ROOTFS}/etc/slackpkg/slackpkg.conf" + sed -i 's/POSTINST=on/POSTINST=off/' "${ROOTFS}/etc/slackpkg/slackpkg.conf" + sed -i 's/SPINNING=on/SPINNING=off/' "${ROOTFS}/etc/slackpkg/slackpkg.conf" + sed -i 's/WGETFLAGS="--passive-ftp/& --no-verbose/' \ + "${ROOTFS}/etc/slackpkg/slackpkg.conf" + fi + + # Configure the slackpkg mirror + [[ -f "${ROOTFS}/etc/slackpkg/mirrors" ]] && \ + printf '%s\n' "${SLACKPKG_MIRROR}" >> "${ROOTFS}/etc/slackpkg/mirrors" + + # slackpkg requires an explicit opt-in for -current + if [[ "${VERSION}" == "current" ]]; then + mkdir -p "${ROOTFS}/var/lib/slackpkg" + touch "${ROOTFS}/var/lib/slackpkg/current" + fi + + # Set a predictable locale for all chroot operations. + # Without this, perl emits locale warnings when the host locale + # (e.g. en_US.UTF-8) is not installed in the minimal rootfs. + export LC_ALL=C + + # Rebuild the CA certificate database inside the rootfs. + # Without this, wget cannot verify Let's Encrypt certificates and every + # HTTPS download by slackpkg fails with an SSL error. That SSL error in + # turn causes the interactive "do you want to import the GPG key?" prompt + # even when -batch=on -default_answer=y is set. + _log "Refreshing CA certificate database..." + chroot "${ROOTFS}" /usr/sbin/update-ca-certificates --fresh 1>/dev/null || _warn "CA certificate update failed; SSL verification may not work correctly." + + # Import the Slackware GPG key and update the package list. + # 'yes YES |' feeds a pre-emptive answer to the interactive import prompt + # as a safety net; with the CA database now valid the prompt should not + # appear because the download succeeds on the first attempt. + _log "Importing Slackware GPG key via slackpkg..." + # Check slackpkg's own exit status, not the pipeline's: 'yes' takes SIGPIPE + # when slackpkg exits first, which can make the pipeline non-zero even on + # success (false "GPG update failed" warning). Wrapping in 'if' exempts the + # pipeline from set -e/pipefail so we can read PIPESTATUS[1] (slackpkg's own + # status) instead of the SIGPIPE-poisoned pipeline status. + local gpg_rc + if yes YES | chroot "${ROOTFS}" /usr/sbin/slackpkg -batch=on -default_answer=y update gpg + then gpg_rc="${PIPESTATUS[1]}"; else gpg_rc="${PIPESTATUS[1]}"; fi + [[ "${gpg_rc}" -eq 0 ]] || _warn "slackpkg GPG update failed; check the output above." + + _log "Updating slackpkg package list..." + chroot "${ROOTFS}" /usr/sbin/slackpkg -batch=on -default_answer=y update || _warn "slackpkg update failed; cache will build on first container run." + + # --- Clean up --- + _log "Cleaning up rootfs..." + rm -f "${ROOTFS}"/boot/* + rm -f "${ROOTFS}"/tmp/[A-Za-z]* + rm -f "${ROOTFS}"/var/mail/* + rm -rf "${ROOTFS}"/dev/* + rm -rf "${ROOTFS}"/usr/share/locale/* + rm -rf "${ROOTFS}"/usr/info/* + rm -rf "${ROOTFS}"/usr/man/* + (cd "${ROOTFS}/usr/doc" 2>/dev/null && \ + find . -type d -mindepth 2 -maxdepth 2 | grep -v '/cups-' | \ + xargs rm -rf) || true + rm -rf "${ROOTFS}"/usr/doc/*/html + rm -f "${ROOTFS}"/usr/doc/*/*.{pdf,db,gz,bz2,xz,txt,TXT} 2>/dev/null || true + rm -rf "${ROOTFS}"/usr/share/gtk-doc + rm -rf "${ROOTFS}"/usr/share/help + find "${ROOTFS}"/usr/share/ -type d -name doc | xargs rm -rf + # Keep only the terminfo entries the container actually needs + find "${ROOTFS}"/usr/share/terminfo/ -type f \ + ! -name 'linux' ! -name 'xterm' ! -name 'screen.linux' \ + -delete 2>/dev/null || true + find "${ROOTFS}"/usr/share/terminfo/ -xtype l -delete 2>/dev/null || true + # gpg-agent socket files left behind by slackpkg in the chroot. + # tar cannot archive Unix domain sockets; removing them eliminates + # the "socket ignored" warnings when the image tarball is created. + rm -f "${ROOTFS}"/root/.gnupg/S.* + + # Kernel-specific content has no meaning in a container + rm -rf "${ROOTFS}"/usr/src + rm -rf "${ROOTFS}"/lib/modules + rm -rf "${ROOTFS}"/lib64/modules 2>/dev/null || true + + # --- Build Docker image --- + _log "Creating image tarball..." + tar -C "${ROOTFS}" -czf "${WORKDIR}/rootfs.tar.gz" . + + local BUILD_DATE + BUILD_DATE="$(date -u +%Y-%m-%dT%H:%M:%SZ)" + + cat > "${WORKDIR}/Dockerfile" <<DOCKERFILE +FROM scratch +LABEL maintainer="Eric Hameleers <alien@slackware.com>" +ADD rootfs.tar.gz / +ENV container=docker +ENV PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin +LABEL org.opencontainers.image.created="${BUILD_DATE}" +LABEL org.opencontainers.image.title="Slackware Linux ${VERSION} (x86_64)" +LABEL org.opencontainers.image.description="Slackware Linux ${VERSION} base container image (x86_64)" +LABEL org.opencontainers.image.vendor="Slackware Linux" +LABEL org.opencontainers.image.licenses="GPL-2.0-or-later AND LGPL-2.1-or-later" +LABEL org.opencontainers.image.url="http://www.slackware.com" +LABEL org.opencontainers.image.source="https://forge.slackware.nl/slackware/docker-images" +LABEL org.opencontainers.image.revision="${VERSION}" +LABEL slackware.version="${VERSION}" +LABEL slackware.arch="x86_64" +CMD ["/bin/bash"] +DOCKERFILE + + # Explicit exit checks: build_variant runs in an `if ! (...)` condition + # (see Main), which suppresses `set -e` in this subshell, so a failed build + # would otherwise push a nonexistent tag and log "Done". + _log "Building and pushing ${TAG}..." + if ! docker build --no-cache -t "${TAG}" "${WORKDIR}"; then + _warn "build failed for ${TAG}; not pushing." + return 1 + fi + if ! docker push "${TAG}"; then + _warn "push failed for ${TAG}." + return 1 + fi + _log "=== Done: ${TAG} ===" +} + +# ============================================================================ +# Main +# ============================================================================ + +rc=0 +for VERSION in "${BUILD_VARIANTS[@]}"; do + if changelog_changed "${VERSION}"; then + if ! ( build_variant "${VERSION}" ); then + _warn "variant ${VERSION} failed; continuing." + rc=1 + fi + fi +done +exit "${rc}" diff --git a/image-builder/build-full-image.sh b/image-builder/build-full-image.sh new file mode 100755 index 0000000..a2fdb18 --- /dev/null +++ b/image-builder/build-full-image.sh @@ -0,0 +1,245 @@ +#!/bin/bash +# +# Copyright (C) 2026 Danilo M. <danix@danix.xyz> +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# build-full-image.sh — sbo-full:{ver} FROM sbo-base:{ver}, all series. +# Adapted from forge slackware/docker-images. No root needed. +set -euo pipefail +HERE="$(cd "$(dirname "$0")" && pwd)" +source "${HERE}/config" +LOG_TAG=full +source "${HERE}/lib.sh" + +BASE_IMAGE="${REGISTRY}/sbo-base" +FULL_IMAGE="${REGISTRY}/sbo-full" +DIGEST_LABEL="sbo.full.base-digest" + +# ============================================================================ +# Argument parsing +# ============================================================================ + +FORCE=false +OPT_VERSION="" +while [[ $# -gt 0 ]]; do + case "$1" in + --version) OPT_VERSION="$2"; shift 2 ;; + --force) FORCE=true; shift ;; + *) _err "unknown argument: $1" ;; + esac +done +if [[ -n "${OPT_VERSION}" ]]; then + BUILD_VARIANTS=("${OPT_VERSION}") +else + BUILD_VARIANTS=("${VARIANTS[@]}") +fi + +# ============================================================================ +# Tag derivation +# ============================================================================ + +# derive_tags VERSION +# Sets globals BASE_TAG and FULL_TAG for the given variant. +derive_tags() { + local VERSION="$1" + BASE_TAG="${BASE_IMAGE}:${VERSION}" + FULL_TAG="${FULL_IMAGE}:${VERSION}" +} + +# ============================================================================ +# Rebuild detection +# +# After pulling the latest base image, its digest is compared with the +# digest stored in the DIGEST_LABEL on the current full image. If +# they differ the base image has been updated and the full must be +# rebuilt to incorporate slackpkg package updates shipped with the new base. +# ============================================================================ + +needs_rebuild() { + local BASE_TAG="$1" FULL_TAG="$2" + + if [[ "${FORCE}" == "true" ]]; then + _log " --force specified; rebuilding unconditionally." + return 0 + fi + + # Digest of the base image we just pulled + local BASE_DIGEST + BASE_DIGEST=$(docker inspect --format '{{index .RepoDigests 0}}' \ + "${BASE_TAG}" 2>/dev/null || echo "") + if [[ -z "${BASE_DIGEST}" ]]; then + _warn " Could not determine base image digest; rebuilding to be safe." + return 0 + fi + + # Digest recorded in the full image's label (empty if image doesn't exist) + local RECORDED_DIGEST + RECORDED_DIGEST=$(docker inspect \ + --format "{{index .Config.Labels \"${DIGEST_LABEL}\"}}" \ + "${FULL_TAG}" 2>/dev/null || echo "") + + if [[ "${BASE_DIGEST}" == "${RECORDED_DIGEST}" ]]; then + _log " Base image unchanged (${BASE_DIGEST:0:40}...); skipping." + return 1 + fi + + _log " Base image changed; rebuilding." + _log " Was : ${RECORDED_DIGEST:0:60}..." + _log " Now : ${BASE_DIGEST:0:60}..." + return 0 +} + +# ============================================================================ +# Build one variant +# ============================================================================ + +build_variant() { + local VERSION="$1" + local REPO_KEY="slackware64-${VERSION}" + require_mount "${VERSION}" # NFS must be up: we serve it over HTTP below + derive_tags "${VERSION}" + + _log "=== ${FULL_TAG} ===" + + # Pull the latest base image so digest comparison is against the current + # state of the registry, not a stale local cache. + _log " Pulling base image ${BASE_TAG}..." + docker pull "${BASE_TAG}" + + if ! needs_rebuild "${BASE_TAG}" "${FULL_TAG}"; then + return 0 + fi + + # Record the base image digest for the label + local BASE_DIGEST + BASE_DIGEST=$(docker inspect --format '{{index .RepoDigests 0}}' "${BASE_TAG}") + + local WORKDIR + WORKDIR="$(mktemp -d /tmp/slackware-full.XXXXXX)" + + # Serve the NFS mirror over HTTP on the docker bridge for the duration of + # the build, so slackpkg in the build container fetches packages from LAN + # instead of the internet. Killed (with WORKDIR cleanup) on any exit. + local MIRROR_ROOT; MIRROR_ROOT="$(strip_scheme "$(mirror_path "${MIRROR}" "${REPO_KEY}")")" + [[ -f "${MIRROR_ROOT}/PACKAGES.TXT" ]] \ + || _err "mirror root has no PACKAGES.TXT: ${MIRROR_ROOT}" + python3 -m http.server "${HTTP_MIRROR_PORT}" \ + --bind "${HTTP_MIRROR_HOST}" --directory "${MIRROR_ROOT}" \ + >/dev/null 2>&1 & + local HTTP_PID=$! + # shellcheck disable=SC2064 + trap "kill ${HTTP_PID} 2>/dev/null; rm -rf '${WORKDIR}'" EXIT + # Wait for the server to accept connections before building. + local i + for i in 1 2 3 4 5 6 7 8 9 10; do + curl -sf -o /dev/null "http://${HTTP_MIRROR_HOST}:${HTTP_MIRROR_PORT}/PACKAGES.TXT" && break + [[ $i -eq 10 ]] && _err "HTTP mirror did not come up on ${HTTP_MIRROR_HOST}:${HTTP_MIRROR_PORT}" + sleep 0.5 + done + + # Generate the Dockerfile for this variant + cat > "${WORKDIR}/Dockerfile" <<DOCKERFILE +FROM ${BASE_TAG} +LABEL maintainer="Eric Hameleers <alien@slackware.com>" + +# Import the Slackware GPG key against THIS mirror first: without it slackpkg +# cannot verify CHECKSUMS.md5 and silently installs almost nothing (only the +# packages already in base). 'yes YES |' answers the import prompt. +RUN yes YES | LC_ALL=C slackpkg -batch=on -default_answer=y update gpg + +# Update the slackpkg package list before installing so that we always +# get the versions current at build time, not the versions cached in the +# base image. +RUN LC_ALL=C slackpkg -batch=on -default_answer=y update + +# Download and install each individual package (thanks aclemons): +RUN sed -i 's/DOWNLOAD_ALL=on/DOWNLOAD_ALL=off/' /etc/slackpkg/slackpkg.conf + +# We don't care about a big firmware package: +RUN mv /etc/slackpkg/blacklist{,.keep} && \\ + echo kernel-firmware > /etc/slackpkg/blacklist + +# Install the full Slackware: +# All packages are installed in a single RUN layer to keep the image lean. +RUN export LC_ALL=C && \\ + for series in a ap d e f k kde l n t tcl x xap xfce y ; do \\ + slackpkg -delall=on -batch=on -default_answer=y install "\$series"/* ; done + +# And back to defaults: +RUN sed -i 's/DOWNLOAD_ALL=off/DOWNLOAD_ALL=on/' /etc/slackpkg/slackpkg.conf +RUN mv /etc/slackpkg/blacklist{.keep,} + + +# Delete package cache: +RUN rm -rf /var/cache/packages/* + +# Rebuild the linker cache after adding new libraries +RUN /sbin/ldconfig + +LABEL org.opencontainers.image.created="$(date -u +%Y-%m-%dT%H:%M:%SZ)" +LABEL org.opencontainers.image.title="Slackware Linux ${VERSION} full (x86_64)" +LABEL org.opencontainers.image.description="Slackware Linux ${VERSION} full image with all packages installed (x86_64)" +LABEL org.opencontainers.image.vendor="Slackware Linux" +LABEL org.opencontainers.image.licenses="GPL-2.0-or-later AND LGPL-2.1-or-later" +LABEL org.opencontainers.image.url="http://www.slackware.com" +LABEL org.opencontainers.image.source="https://forge.slackware.nl/slackware/docker-images" +LABEL org.opencontainers.image.revision="${VERSION}" +LABEL slackware.version="${VERSION}" +LABEL slackware.arch="x86_64" +LABEL ${DIGEST_LABEL}="${BASE_DIGEST}" +CMD ["/bin/bash"] +DOCKERFILE + + local BUILD_FLAGS=() + [[ "${FORCE}" == "true" ]] && BUILD_FLAGS+=(--no-cache) + + # Explicit exit checks: build_variant runs in an `if ! (...)` condition + # (see Main), which suppresses `set -e` inside this subshell, so a failed + # docker build would otherwise fall through to push a nonexistent tag and + # log "Done". Check each step and return non-zero on failure. + _log " Building ${FULL_TAG}..." + if ! docker build "${BUILD_FLAGS[@]}" \ + -t "${FULL_TAG}" \ + "${WORKDIR}"; then + _warn " build failed for ${FULL_TAG}; not pushing." + return 1 + fi + + _log " Pushing ${FULL_TAG}..." + if ! docker push "${FULL_TAG}"; then + _warn " push failed for ${FULL_TAG}." + return 1 + fi + + _log "=== Done: ${FULL_TAG} ===" +} + +# ============================================================================ +# Main +# ============================================================================ + +# A --force build passes --no-cache, so the stale build cache is dead weight: +# it still loads the containerd snapshotter's lease/lock bookkeeping and has +# raced the layer export ("failed to open writer: ref ... locked ... unavailable"). +# Prune it first to shrink that surface. Best-effort; never fail the build on it. +if [[ "${FORCE}" == "true" ]]; then + _log " --force: pruning build cache before rebuild" + docker builder prune -f >/dev/null 2>&1 || _warn " builder prune failed (ignored)" +fi + +rc=0 +for VERSION in "${BUILD_VARIANTS[@]}"; do + if ! ( build_variant "${VERSION}" ); then + _warn "variant ${VERSION} failed; continuing." + rc=1 + fi +done +exit "${rc}" diff --git a/image-builder/build-sbo-testbuild.sh b/image-builder/build-sbo-testbuild.sh new file mode 100755 index 0000000..a246b79 --- /dev/null +++ b/image-builder/build-sbo-testbuild.sh @@ -0,0 +1,143 @@ +#!/bin/bash +# +# Copyright (C) 2026 Danilo M. <danix@danix.xyz> +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# build-sbo-testbuild.sh — sbo-testbuild:{ver} FROM sbo-full:{ver}. +# Adds sbopkg + sbo-maintainer-tools from prebuilt .txz in PKGDIR. +# Rebuilds when the full image OR the .txz set changes. +set -euo pipefail +HERE="$(cd "$(dirname "$0")" && pwd)" +source "${HERE}/config" +LOG_TAG=testbuild +source "${HERE}/lib.sh" + +FULL_IMAGE="${REGISTRY}/sbo-full" +TB_IMAGE="${REGISTRY}/sbo-testbuild" +DIGEST_LABEL="sbo.testbuild.full-digest" +PKGS_LABEL="sbo.testbuild.pkgs-hash" + +FORCE=false +OPT_VERSION="" +while [[ $# -gt 0 ]]; do + case "$1" in + --version) OPT_VERSION="$2"; shift 2 ;; + --force) FORCE=true; shift ;; + *) _err "unknown argument: $1" ;; + esac +done +if [[ -n "${OPT_VERSION}" ]]; then + BUILD_VARIANTS=("${OPT_VERSION}") +else + BUILD_VARIANTS=("${VARIANTS[@]}") +fi + +# Verify the required packages (*.t?z) are present before doing anything. +require_pkgs() { + [[ -d "${PKGDIR}" ]] || _err "PKGDIR missing: ${PKGDIR}" + compgen -G "${PKGDIR}/sbopkg-*.t?z" >/dev/null \ + || _err "no sbopkg-*.t?z in ${PKGDIR}" + compgen -G "${PKGDIR}/sbo-maintainer-tools-*.t?z" >/dev/null \ + || _err "no sbo-maintainer-tools-*.t?z in ${PKGDIR}" +} + +# needs_rebuild FULL_TAG TB_TAG PKGS_HASH -> 0 if rebuild needed +needs_rebuild() { + local full_tag="$1" tb_tag="$2" pkgs_hash="$3" + [[ "${FORCE}" == "true" ]] && { _log " --force."; return 0; } + + local full_digest + full_digest=$(docker inspect --format '{{index .RepoDigests 0}}' \ + "${full_tag}" 2>/dev/null || echo "") + [[ -n "${full_digest}" ]] || { _warn " no full digest; rebuilding."; return 0; } + + local rec_digest rec_pkgs + rec_digest=$(docker inspect \ + --format "{{index .Config.Labels \"${DIGEST_LABEL}\"}}" \ + "${tb_tag}" 2>/dev/null || echo "") + rec_pkgs=$(docker inspect \ + --format "{{index .Config.Labels \"${PKGS_LABEL}\"}}" \ + "${tb_tag}" 2>/dev/null || echo "") + + if [[ "${full_digest}" == "${rec_digest}" && "${pkgs_hash}" == "${rec_pkgs}" ]]; then + _log " full image and .txz set unchanged; skipping." + return 1 + fi + _log " full image or .txz set changed; rebuilding." + return 0 +} + +build_variant() { + local VERSION="$1" + local FULL_TAG="${FULL_IMAGE}:${VERSION}" + local TB_TAG="${TB_IMAGE}:${VERSION}" + _log "=== ${TB_TAG} ===" + + _log " Pulling ${FULL_TAG}..." + docker pull "${FULL_TAG}" + + local PKGS_HASH; PKGS_HASH="$(txz_hash "${PKGDIR}")" + + if ! needs_rebuild "${FULL_TAG}" "${TB_TAG}" "${PKGS_HASH}"; then + return 0 + fi + + local FULL_DIGEST + FULL_DIGEST=$(docker inspect --format '{{index .RepoDigests 0}}' "${FULL_TAG}" 2>/dev/null || echo "") + + local WORKDIR; WORKDIR="$(mktemp -d /tmp/sbo-testbuild.XXXXXX)" + # shellcheck disable=SC2064 + trap "rm -rf '${WORKDIR}'" EXIT + mkdir -p "${WORKDIR}/pkgs" + cp "${PKGDIR}"/sbopkg-*.t?z "${PKGDIR}"/sbo-maintainer-tools-*.t?z "${WORKDIR}/pkgs/" + + cat > "${WORKDIR}/Dockerfile" <<DOCKERFILE +FROM ${FULL_TAG} +LABEL maintainer="danix <danix@danix.xyz>" +COPY pkgs/ /tmp/pkgs/ +RUN installpkg /tmp/pkgs/* && rm -rf /tmp/pkgs +RUN /sbin/ldconfig +LABEL org.opencontainers.image.created="$(date -u +%Y-%m-%dT%H:%M:%SZ)" +LABEL org.opencontainers.image.title="SBo test-build env ${VERSION} (x86_64)" +LABEL org.opencontainers.image.description="Slackware ${VERSION} full + sbopkg + sbo-maintainer-tools" +LABEL slackware.version="${VERSION}" +LABEL ${DIGEST_LABEL}="${FULL_DIGEST}" +LABEL ${PKGS_LABEL}="${PKGS_HASH}" +CMD ["/bin/bash"] +DOCKERFILE + + local BUILD_FLAGS=() + [[ "${FORCE}" == "true" ]] && BUILD_FLAGS+=(--no-cache) + # Explicit exit checks: build_variant runs in an `if ! (...)` condition + # (see Main), which suppresses `set -e` in this subshell, so a failed build + # would otherwise push a nonexistent tag and log "Done". + _log " Building ${TB_TAG}..." + if ! docker build "${BUILD_FLAGS[@]}" -t "${TB_TAG}" "${WORKDIR}"; then + _warn " build failed for ${TB_TAG}; not pushing." + return 1 + fi + _log " Pushing ${TB_TAG}..." + if ! docker push "${TB_TAG}"; then + _warn " push failed for ${TB_TAG}." + return 1 + fi + _log "=== Done: ${TB_TAG} ===" +} + +require_pkgs +rc=0 +for VERSION in "${BUILD_VARIANTS[@]}"; do + if ! ( build_variant "${VERSION}" ); then + _warn "variant ${VERSION} failed; continuing." + rc=1 + fi +done +exit "${rc}" diff --git a/image-builder/config b/image-builder/config new file mode 100644 index 0000000..8b52313 --- /dev/null +++ b/image-builder/config @@ -0,0 +1,15 @@ +# Shared configuration for the sbo-testbuild image builder. +# Sourced by bootstrap.sh, build-full-image.sh, build-sbo-testbuild.sh, tests. + +REGISTRY="docker.noland.dnx:5000" +MIRROR="file:///mnt/nas" +VARIANTS=(current 15.0) # x86_64 only for now +PKGDIR="/opt/sbo-testbuild/pkgs" # sbopkg + sbo-maintainer-tools .txz +HASH_DIR="/var/cache/sbo-testbuild" # ChangeLog hashes for rebuild gating + +# During the full-image build, packages are served to the build container off +# the local NFS mirror (MIRROR) over HTTP on the docker bridge, so slackpkg +# fetches from LAN instead of the internet. The bridge-gateway IP is what the +# build container sees as the host; the port is ephemeral to this host. +HTTP_MIRROR_HOST="172.17.0.1" # docker0 bridge gateway (host from container) +HTTP_MIRROR_PORT="8099" diff --git a/image-builder/lib.sh b/image-builder/lib.sh new file mode 100644 index 0000000..dfb076b --- /dev/null +++ b/image-builder/lib.sh @@ -0,0 +1,84 @@ +# +# Copyright (C) 2026 Danilo M. <danix@danix.xyz> +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# Shared helpers for the image builder. Source, do not execute. +# Sourcing must have no side effects beyond defining functions. + +_log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] [${LOG_TAG:-image-builder}] $*"; } +_warn() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] [${LOG_TAG:-image-builder}] WARNING: $*" >&2; } +_err() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] [${LOG_TAG:-image-builder}] ERROR: $*" >&2; exit 1; } + +# mirror_path MIRROR RELPATH +# Join a MIRROR (file:///... or http(s)://...) with a relative path. +mirror_path() { + local mirror="$1" rel="$2" + printf '%s/%s' "${mirror%/}" "${rel#/}" +} + +# is_local_mirror MIRROR -> 0 if file:// or bare absolute path, else 1 +is_local_mirror() { + case "$1" in + file://*) return 0 ;; + http://*|https://*) return 1 ;; + /*) return 0 ;; + *) return 1 ;; + esac +} + +# strip_scheme URL -> filesystem path for a file:// or bare-local URL +strip_scheme() { + local u="$1" + case "$u" in + file://*) printf '%s' "${u#file://}" ;; + *) printf '%s' "$u" ;; + esac +} + +# fetch SRC DEST +# Copy SRC (a full mirror URL) to DEST. Local mirror -> cp, http -> curl. +# Returns non-zero on failure; caller decides whether that is fatal. +fetch() { + local src="$1" dest="$2" + if is_local_mirror "$src"; then + local path; path="$(strip_scheme "$src")" + [[ -f "$path" ]] || return 1 + cp "$path" "$dest" + else + curl -sSfL --connect-timeout 60 "$src" -o "$dest" + fi +} + +# txz_hash DIR +# Stable hash of the set of Slackware package files (*.t?z) in DIR (names + +# contents), order +# independent. Empty/absent dir -> empty string. +txz_hash() { + local dir="$1" + [[ -d "$dir" ]] || { printf ''; return 0; } + local files; files=$(find "$dir" -maxdepth 1 -name '*.t?z' -type f | sort) + [[ -n "$files" ]] || { printf ''; return 0; } + # Hash sorted "name sha256" lines so reordering does not change output. + while IFS= read -r f; do + printf '%s %s\n' "$(basename "$f")" "$(sha256sum "$f" | cut -d' ' -f1)" + done <<< "$files" | sha256sum | cut -d' ' -f1 +} + +# require_mount VERSION +# Assert the NAS tree for VERSION is mounted (file:// mirror only). +# x86_64 tree dir is slackware64-${VERSION}. +require_mount() { + local version="$1" + is_local_mirror "$MIRROR" || return 0 # http mirror: nothing to check + local base; base="$(strip_scheme "$MIRROR")" + local dir="${base}/slackware64-${version}" + [[ -d "$dir" ]] || _err "NAS tree not mounted: ${dir}" +} diff --git a/image-builder/test-image-builder.sh b/image-builder/test-image-builder.sh new file mode 100755 index 0000000..525ea8f --- /dev/null +++ b/image-builder/test-image-builder.sh @@ -0,0 +1,80 @@ +#!/bin/bash +# +# Copyright (C) 2026 Danilo M. <danix@danix.xyz> +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# Assert-based self-check for lib.sh pure logic. No docker, no network. +# Run: bash test-image-builder.sh +set -u +cd "$(dirname "$0")" +source ./lib.sh + +pass=0 fail=0 +check() { # DESC EXPECTED ACTUAL + if [[ "$2" == "$3" ]]; then pass=$((pass+1)); + else fail=$((fail+1)); echo "FAIL: $1"; echo " expected: [$2]"; echo " actual: [$3]"; fi +} +check_rc() { # DESC EXPECTED_RC ACTUAL_RC + if [[ "$2" == "$3" ]]; then pass=$((pass+1)); + else fail=$((fail+1)); echo "FAIL: $1 (rc expected $2 got $3)"; fi +} + +tmp="$(mktemp -d)" +trap 'rm -rf "$tmp"' EXIT + +# --- mirror_path --- +check "mirror_path trailing slash" \ + "file:///mnt/nas/slackware64-current/PACKAGES.TXT" \ + "$(mirror_path 'file:///mnt/nas/' 'slackware64-current/PACKAGES.TXT')" +check "mirror_path no trailing slash" \ + "file:///mnt/nas/x/y" "$(mirror_path 'file:///mnt/nas' '/x/y')" + +# --- is_local_mirror --- +is_local_mirror "file:///mnt/nas"; check_rc "is_local file://" 0 $? +is_local_mirror "/mnt/nas"; check_rc "is_local bare abs" 0 $? +is_local_mirror "https://x/y"; check_rc "is_local https" 1 $? + +# --- strip_scheme --- +check "strip_scheme file://" "/mnt/nas" "$(strip_scheme 'file:///mnt/nas')" +check "strip_scheme bare" "/mnt/nas" "$(strip_scheme '/mnt/nas')" + +# --- fetch: local file resolves --- +echo hello > "$tmp/src.txt" +fetch "file://$tmp/src.txt" "$tmp/dst.txt"; check_rc "fetch local ok" 0 $? +check "fetch local content" "hello" "$(cat "$tmp/dst.txt" 2>/dev/null)" + +# --- fetch: missing local file errors --- +fetch "file://$tmp/nope.txt" "$tmp/x.txt"; check_rc "fetch local missing" 1 $? + +# --- txz_hash: stable + content-sensitive --- +mkdir -p "$tmp/pkgs" +printf a > "$tmp/pkgs/sbopkg-1.txz" +printf b > "$tmp/pkgs/tools-1.txz" +h1="$(txz_hash "$tmp/pkgs")" +h2="$(txz_hash "$tmp/pkgs")" +check "txz_hash stable" "$h1" "$h2" +printf c > "$tmp/pkgs/tools-1.txz" # change content +h3="$(txz_hash "$tmp/pkgs")" +check_rc "txz_hash changes on content" 0 "$([[ "$h1" != "$h3" ]] && echo 0 || echo 1)" +check "txz_hash empty dir" "" "$(txz_hash "$tmp/empty-nope")" + +# --- require_mount: missing mount errors (subshell to catch _err exit) --- +MIRROR="file://$tmp/no-such-root" +( require_mount current ) 2>/dev/null; check_rc "require_mount missing" 1 $? +mkdir -p "$tmp/mnt/slackware64-current" +MIRROR="file://$tmp/mnt" +( require_mount current ) 2>/dev/null; check_rc "require_mount present" 0 $? +MIRROR="https://example/x" +( require_mount current ) 2>/dev/null; check_rc "require_mount http skips" 0 $? + +echo "----" +echo "PASS: $pass FAIL: $fail" +[[ "$fail" -eq 0 ]] diff --git a/install.sh b/install.sh new file mode 100755 index 0000000..8be02f0 --- /dev/null +++ b/install.sh @@ -0,0 +1,41 @@ +#!/bin/bash +# +# Copyright (C) 2026 Danilo M. <danix@danix.xyz> +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# Install the test-build CLI into ~/bin (override with BINDIR=...). Copies the +# file, so re-run after editing. The image-builder scripts are NOT installed: +# they run on the docker host from their own checkout. +# +# ./install.sh install to ~/bin +# BINDIR=/usr/local/bin ./install.sh +# ./install.sh --uninstall remove it + +set -euo pipefail +HERE="$(cd "$(dirname "$0")" && pwd)" +BINDIR="${BINDIR:-$HOME/bin}" +PROG=test-build + +if [[ "${1:-}" == "--uninstall" ]]; then + rm -f "$BINDIR/$PROG" && echo "removed $BINDIR/$PROG" + exit 0 +fi + +[[ -f "$HERE/$PROG" ]] || { echo "$PROG not found in $HERE" >&2; exit 1; } +mkdir -p "$BINDIR" +install -m 0755 "$HERE/$PROG" "$BINDIR/$PROG" +echo "installed $BINDIR/$PROG" + +case ":$PATH:" in + *":$BINDIR:"*) ;; + *) echo "note: $BINDIR is not in \$PATH; add it (e.g. in ~/.bashrc):" + echo " export PATH=\"$BINDIR:\$PATH\"" ;; +esac diff --git a/overrides.example b/overrides.example new file mode 100644 index 0000000..e9d89e9 --- /dev/null +++ b/overrides.example @@ -0,0 +1,16 @@ +# current-vs-stable dep overrides for test-build. +# Copy to ~/.config/sbo-testbuild/overrides and edit. Applied only when +# targeting -current (15.0 is the SBo baseline). One rule per line. +# +# drop: <prog> dep is a 15.0-only SBo package whose payload the +# -current base already ships; skipped at resolution +# (not built, not flagged UNMET). e.g. rust-opt: +# 15.0 needs the SBo rust-opt, current has rust in base. +# rename: <old> -> <new> dep was renamed on -current +# fetch: <prog> dep was removed from the -current tree; pull via sbopkg +# +# Start light: add rules only when a real package needs them. + +# drop: rust-opt +# rename: python3-foo -> foo +# fetch: somelib diff --git a/test-build b/test-build new file mode 100755 index 0000000..eb4009f --- /dev/null +++ b/test-build @@ -0,0 +1,876 @@ +#!/bin/bash +# +# Copyright (C) 2026 Danilo M. <danix@danix.xyz> +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# +# test-build - verify an already-published SBo package still builds on a target +# Slackware version inside a throwaway docker container. Resolves + builds its +# SBo deps from the local tree, caches built deps per image digest, reports +# per-package status and lints the target. +# +# Dependency-resolution, cache, and summary logic are adapted from sbo-batch-test +# (github: danixland). The overlay chroot is replaced by a docker container: the +# container IS the disposable environment, so no overlayfs. +# +# No em dashes in prose by author convention. + +# ============================================================================= +# CONFIG (do not edit here; real values live in the external config file) +# ============================================================================= +SBO_TREE_CURRENT="" +SBO_TREE_STABLE="" +IMAGE_CURRENT="" +IMAGE_STABLE="" +LOG_ROOT="/var/log/sbo-test-build" +PKG_CACHE="" + +TB_CONFIG="${TB_CONFIG:-$HOME/.config/sbo-testbuild/config}" +if [[ -f "$TB_CONFIG" ]]; then + # shellcheck disable=SC1090 + source "$TB_CONFIG" +fi + +TB_OVERRIDES="${TB_OVERRIDES:-$HOME/.config/sbo-testbuild/overrides}" + +# ============================================================================= +set -uo pipefail +# Not -e: a package build failing is a handled outcome, not a script crash. + +# ---- flags / globals -------------------------------------------------------- +USE_COLOR=1 # --no-color or non-TTY disables +DRY_RUN=0 # --dry-run: resolve + print order, do not build +ASSUME_YES=0 # --yes: skip the confirm prompt (still prints the order) +USE_CACHE=1 # --no-cache disables the dep cache for one run +KEEP_TARGET=0 # --keep: copy the built target package out to KEEP_DIR +VERSION_ID="current" # "current" | "15.0"; set by --stable +TARGET_ARG="" + +ACTIVE_TREE="" # selected SBo tree (by version) +ACTIVE_IMAGE="" # selected image tag (by version) +RUN_DIR="" # timestamped log dir for this run +DEPS_DIR="" # per-run host dir of built dep .txz, mounted into the container +BUILD_OUT="" # per-run host dir where built packages are copied out + +# Status tracking. Keyed by "category/prog". Parallel assoc arrays. +declare -A ST_STATUS=() +declare -A ST_REASON=() +declare -A ST_TIME=() +declare -A ST_README=() + +usage() { + cat <<'EOF' +test-build - verify an SBo package builds on a target Slackware in docker + +USAGE: + test-build [OPTIONS] <program-name> + +OPTIONS: + -h, --help This text. + --stable Target Slackware 15.0 (image + tree). Default is -current. + --dry-run Resolve, apply overrides, print the build order, do not build. + --yes Skip the confirm prompt (the order is still printed first). + --no-cache Rebuild all deps this run, ignore/refresh the cache. + --keep Copy the built target package out to a kept/ dir (path is + printed), so it can be installed on the host. The build is + otherwise throwaway. + --no-color Disable ANSI color (auto-disabled when stdout is not a TTY). +EOF +} + +parse_args() { + while [[ $# -gt 0 ]]; do + case "$1" in + -h|--help) usage; exit 0 ;; + --stable|15.0) VERSION_ID="15.0"; shift ;; + --dry-run) DRY_RUN=1; shift ;; + --yes) ASSUME_YES=1; shift ;; + --no-cache) USE_CACHE=0; shift ;; + --keep) KEEP_TARGET=1; shift ;; + --no-color) USE_COLOR=0; shift ;; + -*) echo "Unknown option: $1" >&2; usage >&2; exit 2 ;; + *) + if [[ -n "$TARGET_ARG" ]]; then + echo "Only one target accepted (got '$TARGET_ARG' and '$1')." >&2 + exit 2 + fi + TARGET_ARG="$1"; shift ;; + esac + done + if [[ -z "$TARGET_ARG" ]]; then + echo "No target given." >&2; usage >&2; exit 2 + fi +} + +init_color() { + if [[ $USE_COLOR -eq 1 && -t 1 ]]; then + C_RED=$'\e[31m'; C_GRN=$'\e[32m'; C_YEL=$'\e[33m'; C_RST=$'\e[0m' + else + C_RED=""; C_GRN=""; C_YEL=""; C_RST="" + fi +} + +# Map VERSION_ID to the active tree + image. No I/O, unit-testable. +select_version_paths() { + if [[ "$VERSION_ID" == "15.0" ]]; then + ACTIVE_TREE="$SBO_TREE_STABLE"; ACTIVE_IMAGE="$IMAGE_STABLE" + else + ACTIVE_TREE="$SBO_TREE_CURRENT"; ACTIVE_IMAGE="$IMAGE_CURRENT" + fi +} + +# require_config: the external config must exist and set the version's paths. +require_config() { + if [[ ! -f "$TB_CONFIG" ]]; then + cat >&2 <<EOF +No config file: $TB_CONFIG +Copy the example and edit it: + + mkdir -p "\$(dirname "$TB_CONFIG")" + cp test-build-config.example "$TB_CONFIG" + \${EDITOR:-vi} "$TB_CONFIG" +EOF + exit 1 + fi + select_version_paths + if [[ -z "$ACTIVE_TREE" || -z "$ACTIVE_IMAGE" ]]; then + echo "Config $TB_CONFIG is missing the tree/image for version '$VERSION_ID'." >&2 + exit 1 + fi + if [[ ! -d "$ACTIVE_TREE" ]]; then + echo "SBo tree for '$VERSION_ID' does not exist: $ACTIVE_TREE" >&2 + exit 1 + fi +} + +# ============================================================================= +# SBo tree lookup +# ============================================================================= +# In this tool there is one active tree per run, but the resolver reads an array +# named SBO_TREE_ROOTS so the ported logic and its self-check match sbo-batch-test. +# main() sets SBO_TREE_ROOTS=("$ACTIVE_TREE") after require_config. +declare -a SBO_TREE_ROOTS=() + +find_slackbuild_dir() { + local prog="$1" root d + for root in "${SBO_TREE_ROOTS[@]}"; do + [[ -d "$root" ]] || continue + for d in "$root"/*/"$prog"; do + if [[ -d "$d" && -f "$d/$prog.info" ]]; then + echo "$d"; return 0 + fi + done + done + return 1 +} + +category_of() { basename "$(dirname "$1")"; } +pkg_key() { echo "$(category_of "$1")/$(basename "$1")"; } + +# Resolve the TARGET to test from CWD or a path (repo-agnostic). The target is +# the SlackBuild you are editing, wherever it lives; its deps come from the +# configured SBo tree (see find_slackbuild_dir), NOT the other way around. +# - a path (absolute, or containing '/', or '.'/'./x'): use that dir directly +# - a bare name: ./<name>/ under CWD, else CWD itself if it IS <name>/ +# Prints the resolved absolute dir on success; returns 1 (with a message) if the +# dir has no matching <name>.info. +resolve_target_dir() { + local arg="$1" dir prog + if [[ "$arg" == /* || "$arg" == .* || "$arg" == */* ]]; then + dir="${arg%/}"; prog="$(basename "$dir")" + elif [[ -f "./$arg/$arg.info" ]]; then + dir="./$arg"; prog="$arg" + elif [[ "$(basename "$PWD")" == "$arg" && -f "./$arg.info" ]]; then + dir="."; prog="$arg" + else + echo "Target '$arg' not found: no ./$arg/$arg.info, and CWD is not $arg/." >&2 + return 1 + fi + if [[ ! -f "$dir/$prog.info" ]]; then + echo "Target dir '$dir' has no $prog.info (expected a SlackBuild package dir)." >&2 + return 1 + fi + ( cd "$dir" && pwd ) # emit absolute path +} + +read_requires() { + local info="$1" + # shellcheck disable=SC1090 + ( set +u; source "$info"; echo "${REQUIRES:-}" ) +} + +version_of() { + local dir="$1" + local info="$dir/$(basename "$dir").info" + [[ -f "$info" ]] || return + local v; v="$(grep -m1 '^VERSION=' "$info" | cut -d'"' -f2)" + echo "$v" +} + +# ============================================================================= +# DEPENDENCY RESOLUTION (topo sort + cycle detection, LOCAL tree only) +# ============================================================================= +declare -a RESOLVED_ORDER=() +declare -A UNMET=() +declare -a CYCLES=() +declare -A HAS_README=() +declare -A _vstate=() +declare -A FETCH_DEPS=() # prog -> 1: resolve via sbopkg in-container + +# Is a prog already present in the container base? Kept as a callback so the +# pure topo logic stays testable. +# +# Populated lazily from the active image's package db on first call: each entry +# in /var/log/packages is named <name>-<ver>-<arch>-<build>, so strip the last +# three dash-fields to get the bare package name and match the SBo token +# against that. TB_BASE_PKGS can be pre-seeded by tests to bypass docker. +declare -A TB_BASE_PKGS=() +TB_BASE_LOADED="${TB_BASE_LOADED:-0}" + +load_base_pkgs() { + [[ "$TB_BASE_LOADED" == "1" ]] && return 0 + TB_BASE_LOADED=1 + [[ -n "$ACTIVE_IMAGE" ]] || return 0 + local entry name + while IFS= read -r entry; do + [[ -n "$entry" ]] || continue + # strip trailing -<ver>-<arch>-<build> + name="${entry%-*-*-*}" + TB_BASE_PKGS["$name"]=1 + done < <(docker run --rm "$ACTIVE_IMAGE" \ + /bin/sh -c 'ls /var/log/packages' 2>/dev/null) +} + +installed_in_base() { + load_base_pkgs + [[ -n "${TB_BASE_PKGS[$1]:-}" ]] +} + +_resolve_visit() { + local dir="$1" parent="$2" + local key; key="$(basename "$dir")" + + if [[ "${_vstate[$dir]:-}" == "1" ]]; then return 0; fi + if [[ "${_vstate[$dir]:-}" == "0" ]]; then + CYCLES+=("cycle involving $key (pulled in via $parent)") + return 1 + fi + _vstate["$dir"]=0 + + local info="$dir/$(basename "$dir").info" + local req tok depdir rc=0 + req="$(read_requires "$info")" + for tok in $req; do + if [[ "$tok" == "%README%" ]]; then + HAS_README["$dir"]=1 + continue + fi + tok="$(apply_rename "$tok")" + # drop: satisfied by the -current base (e.g. a 15.0-only dep like rust-opt + # whose payload ships in current's full-install image). Skip it entirely: + # no recurse, no UNMET, no order entry. Inert on 15.0 (overrides are). + if [[ "${OV_DROP[$tok]:-}" == "1" ]]; then + continue + fi + if [[ "${OV_FETCH[$tok]:-}" == "1" ]]; then + FETCH_DEPS["$tok"]=1 + continue + fi + if depdir="$(find_slackbuild_dir "$tok")"; then + _resolve_visit "$depdir" "$key" || rc=1 + elif installed_in_base "$tok"; then + : + else + UNMET["$tok"]="needed by $key" + rc=1 + fi + done + + _vstate["$dir"]=1 + RESOLVED_ORDER+=("$dir") + return $rc +} + +resolve_target() { + local dir="$1" + RESOLVED_ORDER=() + CYCLES=() + UNMET=() + FETCH_DEPS=() + _vstate=() + _resolve_visit "$dir" "(top)" +} + +# ============================================================================= +# current-vs-stable overrides. Parsed from $TB_OVERRIDES. Applied only when +# targeting -current (15.0 is the SBo baseline, no deltas). +# ============================================================================= +declare -A OV_DROP=() # prog -> 1 +declare -A OV_RENAME=() # old -> new +declare -A OV_FETCH=() # prog -> 1 + +load_overrides() { + OV_DROP=(); OV_RENAME=(); OV_FETCH=() + # 15.0 is the baseline: no overrides. + [[ "$VERSION_ID" == "15.0" ]] && return + [[ -f "$TB_OVERRIDES" ]] || return + local line kind rest + while IFS= read -r line; do + line="${line%%#*}" # strip comments + line="${line#"${line%%[![:space:]]*}"}" # ltrim + [[ -z "$line" ]] && continue + kind="${line%%:*}"; rest="${line#*:}" + kind="${kind//[[:space:]]/}" + rest="${rest#"${rest%%[![:space:]]*}"}" # ltrim value + case "$kind" in + drop) OV_DROP["${rest//[[:space:]]/}"]=1 ;; + fetch) OV_FETCH["${rest//[[:space:]]/}"]=1 ;; + rename) + # rest is "old -> new" + local old new + old="${rest%%->*}"; new="${rest##*->}" + old="${old//[[:space:]]/}"; new="${new//[[:space:]]/}" + [[ -n "$old" && -n "$new" ]] && OV_RENAME["$old"]="$new" ;; + *) echo "WARN: unknown override rule: $line" >&2 ;; + esac + done < "$TB_OVERRIDES" +} + +# Map a dep token through rename rules (identity if no rule). +apply_rename() { + local tok="$1" + echo "${OV_RENAME[$tok]:-$tok}" +} + +# Remove dropped packages from RESOLVED_ORDER in place. +apply_overrides_to_order() { + local d prog keep=() + for d in "${RESOLVED_ORDER[@]}"; do + prog="$(basename "$d")" + [[ "${OV_DROP[$prog]:-}" == "1" ]] && continue + keep+=("$d") + done + # Assign without "${keep[@]:-}": under set -u that fallback yields a + # one-element array holding an empty string when keep is empty (all deps + # dropped), which the build loop would then iterate over as d="". Guard it. + if [[ ${#keep[@]} -eq 0 ]]; then + RESOLVED_ORDER=() + else + RESOLVED_ORDER=("${keep[@]}") + fi +} + +# ============================================================================= +# Dependency cache. Layout: $CACHE_ROOT/<cat>/<prog>/<prog>-<ver>-...txz where +# CACHE_ROOT = $PKG_CACHE/<image-digest> (set per run by resolve_cache_root). +# Key is prog+version. --no-cache (USE_CACHE=0) or empty PKG_CACHE disables. +# ============================================================================= +CACHE_ROOT="" # set by resolve_cache_root once the image digest is known + +# True when the cache is usable this run. +_cache_on() { [[ $USE_CACHE -eq 1 && -n "$PKG_CACHE" && -n "$CACHE_ROOT" ]]; } + +_cache_ver_of() { + local prog="$1" base="$2" + base="${base#"$prog"-}" + echo "${base%%-*}" +} + +# cache_decision <cat> <prog> <version> -> cached | bump:OLD:NEW | new +cache_decision() { + local cat="$1" prog="$2" version="$3" + _cache_on || { echo new; return; } + local dir="$CACHE_ROOT/$cat/$prog" + local f newest="" + for f in "$dir/$prog"-*.t?z; do + [[ -e "$f" ]] || continue + [[ -z "$newest" || "$f" -nt "$newest" ]] && newest="$f" + done + [[ -z "$newest" ]] && { echo new; return; } + local have; have="$(_cache_ver_of "$prog" "$(basename "$newest")")" + if [[ "$have" == "$version" ]]; then echo cached; else echo "bump:$have:$version"; fi +} + +cache_path() { + local cat="$1" prog="$2" version="$3" + _cache_on || return + local dir="$CACHE_ROOT/$cat/$prog" + local f newest="" + for f in "$dir/$prog"-*.t?z; do + [[ -e "$f" ]] || continue + [[ -z "$newest" || "$f" -nt "$newest" ]] && newest="$f" + done + [[ -z "$newest" ]] && return + [[ "$(_cache_ver_of "$prog" "$(basename "$newest")")" == "$version" ]] && echo "$newest" +} + +cache_store() { + local cat="$1" prog="$2" src="$3" + _cache_on || return + local dir="$CACHE_ROOT/$cat/$prog" + mkdir -p "$dir" + rm -f "$dir"/*.t?z + cp -a "$src" "$dir/" +} + +cache_label() { + local dir="$1" is_target="$2" + local cat prog ver dec + cat="$(category_of "$dir")"; prog="$(basename "$dir")"; ver="$(version_of "$dir")" + dec="$(cache_decision "$cat" "$prog" "$ver")" + local label + case "$dec" in + cached) label="cached ($ver)" ;; + bump:*) label="rebuild: ${dec#bump:}"; label="${label/:/ -> }" ;; + *) label="build (new)" ;; + esac + if [[ "$is_target" == "1" ]]; then + case "$dec" in + cached) label="build (cached $ver, rebuilt as target)" ;; + esac + echo "target, $label" + else + echo "$label" + fi +} + +# Compute CACHE_ROOT from the image's digest, namespaced by variant. Falls back +# to the tag if the digest cannot be read (still isolates per image reference). +# The dir is "<variant>-<digest>", so an image rebuild changes the digest and +# self-invalidates the cache (deps built against the old base are never reused). +# +# Prune stale digests of THIS variant on the way in: a base update leaves the +# old <variant>-<olddigest> dir orphaned, so drop it. Scoped to the variant so a +# --current run does not wipe the 15.0 cache (both live caches survive, and +# testing both trees back-to-back still reuses built deps). +resolve_cache_root() { + [[ -z "$PKG_CACHE" ]] && { CACHE_ROOT=""; return; } + local digest + digest="$(docker image inspect --format '{{index .Id}}' "$ACTIVE_IMAGE" 2>/dev/null)" + [[ -z "$digest" ]] && digest="tag-${ACTIVE_IMAGE//[^a-zA-Z0-9._-]/_}" + digest="${digest//[^a-zA-Z0-9._-]/_}" + local name="${VERSION_ID}-${digest}" + CACHE_ROOT="$PKG_CACHE/$name" + mkdir -p "$CACHE_ROOT" + + # drop superseded caches for this variant only + local d + for d in "$PKG_CACHE/${VERSION_ID}-"*; do + [[ -d "$d" ]] || continue + [[ "$(basename "$d")" == "$name" ]] && continue + rm -rf "$d" + done +} + +# Does SlackBuild dir $1 directly require any prog in the dead list (nameref $2)? +# Direct-requires check only; transitive blocking works because run_target +# iterates in topo order, propagating a failure one hop per package. +depends_on_failed() { + local dir="$1"; local -n failed="$2" + local info="$dir/$(basename "$dir").info" + local req tok f + req="$(read_requires "$info")" + for tok in $req; do + [[ "$tok" == "%README%" ]] && continue + for f in "${failed[@]:-}"; do + [[ "$tok" == "$f" ]] && return 0 + done + done + return 1 +} + +# build_one <slackbuild-dir> <is_target 0|1> [container-name] +# Runs the build in a throwaway container. Sets ST_STATUS/ST_REASON/ST_TIME. +# Successful builds copy their package to a host workdir; deps are cached and +# the target is linted. Returns 0 on SUCCESS/CACHED, 1 otherwise. +build_one() { + local dir="$1" is_target="${2:-0}" + local prog cat key + prog="$(basename "$dir")"; cat="$(category_of "$dir")"; key="$cat/$prog" + local logf="$RUN_DIR/${cat}_${prog}.log" + local start; start=$(date +%s) + local version; version="$(version_of "$dir")" + + [[ "${HAS_README[$dir]:-}" == "1" ]] && ST_README["$key"]=1 + + # Dep with a version-matching cached package: installpkg it into the shared + # dep-package dir; no build. The target never takes this path. + if [[ "$is_target" != "1" ]]; then + local cached; cached="$(cache_path "$cat" "$prog" "$version")" + if [[ -n "$cached" ]]; then + cp -a "$cached" "$DEPS_DIR/" + { + echo "===== test-build: $prog (from cache) =====" + echo "cached package: $(basename "$cached")" + } >> "$logf" + ST_TIME["$key"]=$(( $(date +%s) - start )) + ST_STATUS["$key"]="CACHED" + return 0 + fi + fi + + # Build in a container. Mounts: + # $dir -> /sbo/pkg (ro, the SlackBuild) + # $DEPS_DIR -> /sbo/deps (rw, already-built dep .txz to installpkg first) + # $BUILD_OUT -> /sbo/out (rw, where the built package is copied out) + # The in-container script installs any deps present, then builds the target, + # writes a status token to /sbo/out/$prog.status, and copies the package out. + local statf="$BUILD_OUT/$prog.status" + rm -f "$statf" + + # fetch deps (removed from -current tree): let the container's sbopkg build + # them first. FETCH_DEPS is the set collected during resolution. + local fetch_list="" + local fp + for fp in "${!FETCH_DEPS[@]}"; do fetch_list+="$fp "; done + + # -i is required: without it docker does not attach stdin, so `bash -s` reads + # nothing and the heredoc script is silently discarded (exit 0, empty log). + docker run --rm -i \ + -v "$dir":/sbo/pkg:ro \ + -v "$DEPS_DIR":/sbo/deps \ + -v "$BUILD_OUT":/sbo/out \ + -e PROG="$prog" \ + -e FETCH_LIST="$fetch_list" \ + -e IS_TARGET="$is_target" \ + "$ACTIVE_IMAGE" /bin/bash -s >>"$logf" 2>&1 <<'CONTAINER_EOF' +set -uo pipefail +prog="$PROG" +statf="/sbo/out/$prog.status" + +# 0. install already-built dependency packages (order guaranteed by the host). +for d in /sbo/deps/*.t?z; do + [[ -e "$d" ]] || continue + installpkg --terse "$d" || { echo "INSTALL-FAILED (dep $d)"; echo INSTALL-FAILED > "$statf"; exit 1; } +done + +# 0b. fetch-from-SBo deps via sbopkg (removed from the -current tree). +for f in $FETCH_LIST; do + echo "sbopkg-building fetch dep: $f" + sbopkg -B -i "$f" || { echo "BUILD-FAILED (fetch dep $f)"; echo BUILD-FAILED > "$statf"; exit 1; } +done + +# copy the SlackBuild out of the read-only mount so it can write there. +cp -a /sbo/pkg /sbo/build +cd /sbo/build || { echo BUILD-FAILED > "$statf"; exit 1; } +. ./"$prog".info + +export OUTPUT=/sbo/out +mkdir -p "$OUTPUT" + +echo "===== test-build: $prog =====" +echo "PRGNAM=${PRGNAM:-$prog} VERSION=${VERSION:-?} BUILD=${BUILD:-?} TAG=${TAG:-?}" +echo "uname -m: $(uname -m) OUTPUT=$OUTPUT" +echo "REQUIRES=${REQUIRES:-}" +echo "=================================" + +if [ "$(uname -m)" = "x86_64" ] && [ -n "${DOWNLOAD_x86_64:-}" ] && [ "${DOWNLOAD_x86_64}" != "UNSUPPORTED" ] && [ "${DOWNLOAD_x86_64}" != "UNTESTED" ]; then + DL="$DOWNLOAD_x86_64"; MD="$MD5SUM_x86_64" +else + DL="$DOWNLOAD"; MD="$MD5SUM" +fi + +for u in $DL; do + wget -c --tries=3 "$u" || { echo DOWNLOAD-FAILED > "$statf"; exit 1; } +done + +set -- $MD +for u in $DL; do + f="$(basename "$u")" + want="$1"; shift + got="$(md5sum "$f" | cut -d' ' -f1)" + if [ "$got" != "$want" ]; then + echo "MD5 mismatch on $f: want $want got $got" + echo MD5-MISMATCH > "$statf"; exit 1 + fi +done + +# Source /etc/profile.d so dep-provided env is live (google-go-lang sets GOROOT +# + PATH to its go here, rust-opt sets cargo, etc.). This heredoc runs in a +# non-login shell, which does NOT read profile.d, so a bare `go build` would +# otherwise pick the system gccgo instead of the installed google-go-lang. +set +u # profile.d scripts routinely reference unset vars +for pf in /etc/profile.d/*.sh; do + [ -r "$pf" ] && . "$pf" +done +set -u + +chmod +x ./"$prog".SlackBuild +if ! ./"$prog".SlackBuild; then + echo BUILD-FAILED > "$statf"; exit 1 +fi + +pkg="$(ls -t "$OUTPUT"/"$prog"-*.t?z 2>/dev/null | head -n1)" +if [ -z "$pkg" ]; then + echo "No package produced in $OUTPUT" + echo BUILD-FAILED > "$statf"; exit 1 +fi +if ! installpkg --terse "$pkg"; then + echo INSTALL-FAILED > "$statf"; exit 1 +fi +echo "===== installed files: $(basename "$pkg") =====" +pkgname="$(basename "$pkg")"; pkgname="${pkgname%.t?z}" +cat "/var/log/packages/$pkgname" 2>/dev/null || echo "(package db entry not found)" +echo "=================================" + +# lint the target here in the container: sbopkglint is baked into the image and +# runs as root, so it needs no host sudo. Fail-soft: findings never fail the build. +if [ "${IS_TARGET:-0}" = "1" ] && command -v sbopkglint >/dev/null 2>&1; then + echo "===== sbopkglint: $(basename "$pkg") =====" + if sbopkglint "$pkg"; then + echo "LINT-CLEAN" + else + echo "LINT-FINDINGS" + fi + echo "=================================" +fi +echo SUCCESS > "$statf" +CONTAINER_EOF + + local status="BUILD-FAILED" + [[ -f "$statf" ]] && status="$(cat "$statf")" + ST_TIME["$key"]=$(( $(date +%s) - start )) + ST_STATUS["$key"]="$status" + + if [[ "$status" == "SUCCESS" ]]; then + # locate the built package copied to the host workdir + local built newest="" + for built in "$BUILD_OUT/${prog}"-*.t?z; do + [[ -e "$built" ]] || continue + [[ -z "$newest" || "$built" -nt "$newest" ]] && newest="$built" + done + if [[ -n "$newest" ]]; then + if [[ "$is_target" != "1" ]]; then + cache_store "$cat" "$prog" "$newest" + # make the dep available to later builds in this run + cp -a "$newest" "$DEPS_DIR/" + else + # lint ran in-container (see IS_TARGET block); surface its verdict from + # the log so the host summary can show clean/findings. + if grep -q '^LINT-FINDINGS$' "$logf" 2>/dev/null; then + echo " lint: ${C_RED}findings${C_RST} (see $(basename "$logf"))" + elif grep -q '^LINT-CLEAN$' "$logf" 2>/dev/null; then + echo " lint: ${C_GRN}clean${C_RST}" + fi + # --keep: copy the built target package to a durable dir (sibling of the + # logs tree, not inside a throwaway run dir) so it can be installed on + # the host, e.g. to regenerate post-install artifacts. + if [[ $KEEP_TARGET -eq 1 ]]; then + local keepdir; keepdir="$(dirname "$LOG_ROOT")/kept" + mkdir -p "$keepdir" + cp -a "$newest" "$keepdir/" + echo " kept: $keepdir/$(basename "$newest")" + fi + fi + fi + return 0 + fi + ST_REASON["$key"]="see $(basename "$logf")" + return 1 +} + +# confirm_order: print the resolved order (overrides marked) and ask to proceed. +# --yes skips the prompt but the order is still printed. --dry-run never reaches +# here. Returns 0 to proceed, 1 to abort. +confirm_order() { + local target_dir="$1" + echo " build order (${VERSION_ID}):" + local d + for d in "${RESOLVED_ORDER[@]}"; do + local it=0; [[ "$d" == "$target_dir" ]] && it=1 + local rdm=""; [[ "${HAS_README[$d]:-}" == 1 ]] && rdm=" [%README%]" + printf " %-30s %s%s\n" "$(pkg_key "$d")" "$(cache_label "$d" "$it")" "$rdm" + echo "$(pkg_key "$d")" >> "$RUN_DIR/build-order.txt" + done + # note fetch deps (built via sbopkg in-container, not in the order list) + local fp + for fp in "${!FETCH_DEPS[@]}"; do + printf " %-30s %s\n" "$fp" "fetch (sbopkg in container)" + done + [[ $ASSUME_YES -eq 1 ]] && return 0 + local reply + read -rp " Proceed? [Y/n] " reply + [[ -z "$reply" || "$reply" =~ ^[Yy]$ ]] +} + +# run_target <target-slackbuild-dir> +run_target() { + local target_dir="$1" + local tkey; tkey="$(pkg_key "$target_dir")" + + echo + echo "=== Target: $tkey (${VERSION_ID}) ===" + resolve_target "$target_dir" + apply_overrides_to_order + + # Hard resolution failures: report and stop, do not build. + if [[ ${#CYCLES[@]} -gt 0 || ${#UNMET[@]} -gt 0 ]]; then + local why="" + if [[ ${#UNMET[@]} -gt 0 ]]; then + local u + for u in "${!UNMET[@]}"; do why+="unmet:$u(${UNMET[$u]}) "; done + fi + [[ ${#CYCLES[@]} -gt 0 ]] && why+="${CYCLES[*]}" + ST_STATUS["$tkey"]="UNMET-DEP" + ST_REASON["$tkey"]="$why" + echo " resolution failed: $why" + echo " add an override rule ($TB_OVERRIDES) and rerun, or fix the tree." >&2 + return + fi + + if [[ $DRY_RUN -eq 1 ]]; then + echo " build order (dry-run):" + local d + for d in "${RESOLVED_ORDER[@]}"; do + local it=0; [[ "$d" == "$target_dir" ]] && it=1 + local rdm=""; [[ "${HAS_README[$d]:-}" == 1 ]] && rdm=" [%README%]" + printf " %-30s %s%s\n" "$(pkg_key "$d")" "$(cache_label "$d" "$it")" "$rdm" + echo "$(pkg_key "$d")" >> "$RUN_DIR/build-order.txt" + done + local fp + for fp in "${!FETCH_DEPS[@]}"; do + printf " %-30s %s\n" "$fp" "fetch (sbopkg in container)" + done + return + fi + + if ! confirm_order "$target_dir"; then + echo " aborted." + ST_STATUS["$tkey"]="ABORTED" + return + fi + + # Per-run docker workdirs (host side, discarded after the run). + DEPS_DIR="$RUN_DIR/deps"; BUILD_OUT="$RUN_DIR/out" + mkdir -p "$DEPS_DIR" "$BUILD_OUT" + + local d failed_progs=() + for d in "${RESOLVED_ORDER[@]}"; do + local key; key="$(pkg_key "$d")" + local prog; prog="$(basename "$d")" + + if depends_on_failed "$d" failed_progs; then + ST_STATUS["$key"]="BLOCKED-BY-DEP" + ST_REASON["$key"]="blocked by failed dep" + [[ "${HAS_README[$d]:-}" == "1" ]] && ST_README["$key"]=1 + echo " $key: BLOCKED-BY-DEP" + failed_progs+=("$prog") + continue + fi + + local it=0; [[ "$d" == "$target_dir" ]] && it=1 + echo " building $key ..." + if build_one "$d" "$it"; then + echo " $key: ${ST_STATUS[$key]} (${ST_TIME[$key]}s)" + else + echo " $key: ${ST_STATUS[$key]} (${ST_TIME[$key]}s)" + failed_progs+=("$prog") + fi + done +} + +# ============================================================================= +# SUMMARY (ported) +# ============================================================================= +print_summary() { + local total=$SECONDS + local succ=0 fail=0 blocked=0 cached=0 + local summary="$RUN_DIR/summary.log" + + { + echo "test-build run summary" + echo "target: $TARGET_ARG version: $VERSION_ID" + echo + } > "$summary" + + echo + echo "================ SUMMARY ================" + local key + for key in "${!ST_STATUS[@]}"; do + local st="${ST_STATUS[$key]}" rsn="${ST_REASON[$key]:-}" t="${ST_TIME[$key]:-0}" + local rd=""; [[ "${ST_README[$key]:-}" == "1" ]] && rd=" [%README%]" + local col="$C_YEL" + case "$st" in + SUCCESS) col="$C_GRN"; ((succ++)) ;; + CACHED) col="$C_GRN"; ((cached++)) ;; + BLOCKED-BY-DEP|UNMET-DEP|ABORTED) col="$C_YEL"; ((blocked++)) ;; + *) col="$C_RED"; ((fail++)) ;; + esac + printf "%s%-30s %-16s%s %s%s (%ss)\n" "$col" "$key" "$st" "$C_RST" "$rsn" "$rd" "$t" + printf "%-30s %-16s %s%s (%ss)\n" "$key" "$st" "$rsn" "$rd" "$t" >> "$summary" + done + echo "----------------------------------------" + printf "%s%d succeeded%s, %s%d failed%s, %s%d blocked%s, %s%d cached%s, total %ss\n" \ + "$C_GRN" "$succ" "$C_RST" "$C_RED" "$fail" "$C_RST" "$C_YEL" "$blocked" "$C_RST" \ + "$C_GRN" "$cached" "$C_RST" "$total" + echo "logs: $RUN_DIR" + if [[ $fail -eq 0 && $blocked -eq 0 ]]; then + echo "${C_GRN}All green.${C_RST} Safe to build the SBo submission tarball on the host." + fi + { + echo + echo "$succ succeeded, $fail failed, $blocked blocked, $cached cached, total ${total}s" + echo "logs: $RUN_DIR" + } >> "$summary" +} + + +# Ensure the selected image is available locally. It is built by a separate job +# (the image-builder) and pushed to the LAN registry; this script only consumes +# it. If it is not already local, pull it once (ACTIVE_IMAGE is a fully-qualified +# registry ref). A stale local tag is not refreshed here. +# ponytail: pull-if-missing only. If the registry's :current is rebuilt, the +# local copy goes stale silently. Add a --pull force-flag if that bites. +require_image() { + docker image inspect "$ACTIVE_IMAGE" >/dev/null 2>&1 && return 0 + echo "Image not present locally, pulling: $ACTIVE_IMAGE" >&2 + docker pull "$ACTIVE_IMAGE" >&2 && return 0 + cat >&2 <<EOF +Image not found and pull failed: $ACTIVE_IMAGE +It is produced by the image-builder job (full Slackware $VERSION_ID + +sbo-maintainer-tools + sbopkg) and pushed to the LAN registry. Check the +registry is reachable and the tag exists, then rerun. This script does not +build images. +EOF + exit 1 +} + +main() { + parse_args "$@" + init_color + require_config + SBO_TREE_ROOTS=("$ACTIVE_TREE") + load_overrides + + # image + cache namespace (skip the image check on a pure dry-run so the order + # can be inspected without the image present). + if [[ $DRY_RUN -eq 0 ]]; then + command -v docker >/dev/null 2>&1 || { echo "docker not found in PATH." >&2; exit 1; } + require_image + fi + resolve_cache_root + + RUN_DIR="$LOG_ROOT/$(date +%Y-%m-%d_%H-%M-%S)" + mkdir -p "$RUN_DIR" + : > "$RUN_DIR/build-order.txt" + + # The target is the local SlackBuild under test (CWD or a path); its deps are + # resolved from the configured SBo tree during _resolve_visit. + local tdir + if ! tdir="$(resolve_target_dir "$TARGET_ARG")"; then + exit 1 + fi + + run_target "$tdir" + print_summary +} + +main "$@" diff --git a/test-build-config.example b/test-build-config.example new file mode 100644 index 0000000..a07b2c1 --- /dev/null +++ b/test-build-config.example @@ -0,0 +1,18 @@ +# test-build config. Copy to ~/.config/sbo-testbuild/config and edit. +# Override the path with the TB_CONFIG environment variable. + +# Local SBo trees, one per target Slackware version. +SBO_TREE_CURRENT="/home/danix/SBo/current" +SBO_TREE_STABLE="/home/danix/SBo/15.0" + +# Ready image tags (built by a separate job, consumed here by tag). +IMAGE_CURRENT="sbo-testbuild:current" +IMAGE_STABLE="sbo-testbuild:15.0" + +# Where per-run logs land. +LOG_ROOT="/home/danix/.cache/sbo-test-build" + +# Dependency package cache (host dir). Empty disables caching. +# Namespaced internally by image digest, so it self-invalidates on image update +# and keeps current vs 15.0 separate. +PKG_CACHE="/home/danix/.cache/sbo-test-build/pkgcache" diff --git a/test-logic.sh b/test-logic.sh new file mode 100644 index 0000000..da50513 --- /dev/null +++ b/test-logic.sh @@ -0,0 +1,227 @@ +#!/bin/bash +# +# Copyright (C) 2026 Danilo M. <danix@danix.xyz> +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# +# Logic self-check for test-build. Covers the pure, host-side parts: +# dependency resolution, override application, unknown-dep -> UNMET, cache +# decisions, and BLOCKED-BY-DEP propagation. No docker. +# +# Run: bash test-logic.sh +# +set -uo pipefail + +SCRIPT="$(dirname "$0")/test-build" +T=$(mktemp -d) +BASE_DB=$(mktemp -d); mkdir -p "$BASE_DB" + +cleanup() { rm -rf "$T" "$BASE_DB"; } +trap cleanup EXIT + +# Source the script without running main(). Sourcing re-runs the CONFIG block, +# so set test vars AFTER the source. +LIB=$(mktemp) +sed '/^main "\$@"$/d' "$SCRIPT" > "$LIB" +# shellcheck disable=SC1090 +source "$LIB" 2>/dev/null +rm -f "$LIB" + +pass=0; fail=0 +ok() { echo " ok: $1"; ((pass++)); return 0; } +bad() { echo " FAIL: $1"; ((fail++)); return 0; } + +# Placeholder assertion so this file runs before any logic exists. +ok "script sources without executing main" + +# --- version selection ------------------------------------------------------ +SBO_TREE_CURRENT="/trees/current" +SBO_TREE_STABLE="/trees/stable" +IMAGE_CURRENT="sbo-testbuild:current" +IMAGE_STABLE="sbo-testbuild:15.0" + +VERSION_ID="current"; select_version_paths +[[ "$ACTIVE_TREE" == "/trees/current" ]] && ok "current -> current tree" || bad "current tree wrong: [$ACTIVE_TREE]" +[[ "$ACTIVE_IMAGE" == "sbo-testbuild:current" ]] && ok "current -> current image" || bad "current image wrong: [$ACTIVE_IMAGE]" + +VERSION_ID="15.0"; select_version_paths +[[ "$ACTIVE_TREE" == "/trees/stable" ]] && ok "15.0 -> stable tree" || bad "stable tree wrong: [$ACTIVE_TREE]" +[[ "$ACTIVE_IMAGE" == "sbo-testbuild:15.0" ]] && ok "15.0 -> stable image" || bad "stable image wrong: [$ACTIVE_IMAGE]" + +# --- resolution ------------------------------------------------------------- +# Fake SBo tree under one category. mk <prog> "<REQUIRES>". +mk() { mkdir -p "$T/cat/$1"; echo "REQUIRES=\"$2\"" > "$T/cat/$1/$1.info"; } +mk c "" +mk b "c" +mk a "b %README%" +mk d "nonexistentpkg" +mk e "f" +mk f "e" +mk g "b" + +SBO_TREE_ROOTS=("$T") # resolver reads this global (set by select in real runs) + +resolve_target "$T/cat/a" +order=""; for x in "${RESOLVED_ORDER[@]}"; do order+="$(basename "$x") "; done +order="${order% }" +[[ "$order" == "c b a" ]] && ok "topo order c b a" || bad "topo order, got: [$order]" +[[ "${HAS_README[$T/cat/a]:-}" == "1" ]] && ok "%README% recorded" || bad "%README% not recorded" +[[ ${#UNMET[@]} -eq 0 ]] && ok "no false unmet" || bad "unexpected unmet" + +resolve_target "$T/cat/d" +[[ ${#UNMET[@]} -eq 1 ]] && ok "unmet-dep caught" || bad "unmet-dep missed" + +# --- resolve_target_dir (target from CWD/path, NOT the SBo tree) ------------- +# Local package dir, separate from the fake SBo tree above. +LOCALREPO=$(mktemp -d); mkdir -p "$LOCALREPO/mypkg" +echo 'REQUIRES=""' > "$LOCALREPO/mypkg/mypkg.info" +# 1. absolute path +got="$(resolve_target_dir "$LOCALREPO/mypkg" 2>/dev/null)" +[[ "$got" == "$LOCALREPO/mypkg" ]] && ok "target: absolute path" || bad "abs path got [$got]" +# 2. bare name resolved under CWD (./mypkg/) +got="$(cd "$LOCALREPO" && resolve_target_dir "mypkg" 2>/dev/null)" +[[ "$got" == "$LOCALREPO/mypkg" ]] && ok "target: bare name in CWD" || bad "bare-in-cwd got [$got]" +# 3. CWD itself is the package +got="$(cd "$LOCALREPO/mypkg" && resolve_target_dir "mypkg" 2>/dev/null)" +[[ "$got" == "$LOCALREPO/mypkg" ]] && ok "target: CWD is the package" || bad "cwd-is-pkg got [$got]" +# 4. missing -> failure +resolve_target_dir "nosuchpkg" >/dev/null 2>&1 && bad "missing target should fail" || ok "target: missing fails" +rm -rf "$LOCALREPO" + +resolve_target "$T/cat/e" +[[ ${#CYCLES[@]} -ge 1 ]] && ok "cycle caught" || bad "cycle missed" + +# --- installed_in_base ------------------------------------------------------ +# Pre-seed the base pkg set (TB_BASE_LOADED=1 bypasses docker). Names are the +# bare package name; installed_in_base must match a REQUIRES token against it. +TB_BASE_LOADED=1 +TB_BASE_PKGS=([python3]=1 [libfoo]=1) +installed_in_base python3 && ok "base pkg matched" || bad "base pkg not matched" +installed_in_base notthere && bad "absent pkg wrongly matched" || ok "absent pkg not matched" + +# a dep present in the base resolves as met, not UNMET +mk h "python3" +resolve_target "$T/cat/h" +[[ ${#UNMET[@]} -eq 0 ]] && ok "base-provided dep not unmet" || bad "base dep flagged unmet" +TB_BASE_LOADED=0; TB_BASE_PKGS=() + +# --- overrides -------------------------------------------------------------- +VERSION_ID="current" # reset: left at 15.0 by the version-selection checks above +OV=$(mktemp) +cat > "$OV" <<'EOF' +# comment ignored +drop: dropme +rename: oldname -> newname +fetch: fetchme +EOF +TB_OVERRIDES="$OV" +load_overrides + +[[ "${OV_DROP[dropme]:-}" == "1" ]] && ok "override drop parsed" || bad "drop not parsed" +[[ "${OV_RENAME[oldname]:-}" == "newname" ]] && ok "override rename parsed" || bad "rename not parsed" +[[ "${OV_FETCH[fetchme]:-}" == "1" ]] && ok "override fetch parsed" || bad "fetch not parsed" + +# rename maps a token +[[ "$(apply_rename oldname)" == "newname" ]] && ok "apply_rename maps" || bad "apply_rename wrong: [$(apply_rename oldname)]" +[[ "$(apply_rename untouched)" == "untouched" ]] && ok "apply_rename passthrough" || bad "apply_rename mangled untouched" + +# drop satisfies an external (no-dir) dep during resolution: not UNMET, absent +# from the order. mk k requires a dropme that has no dir in the tree. +OV_DROP=([extdrop]=1) +mk k "extdrop" +resolve_target "$T/cat/k" +[[ ${#UNMET[@]} -eq 0 ]] && ok "drop satisfies external dep (not unmet)" || bad "dropped external dep still unmet" +seen=""; for x in "${RESOLVED_ORDER[@]}"; do seen+="$(basename "$x") "; done +[[ "$seen" != *extdrop* ]] && ok "dropped dep absent from order" || bad "dropped dep in order: [$seen]" +load_overrides # restore OV_DROP[dropme] for the order-filter test below + +# drop filters the order (build a fake order of dirs; dropme is removed) +mk dropme "" +mk keepme "" +RESOLVED_ORDER=("$T/cat/dropme" "$T/cat/keepme") +apply_overrides_to_order +kept=""; for x in "${RESOLVED_ORDER[@]}"; do kept+="$(basename "$x") "; done +kept="${kept% }" +[[ "$kept" == "keepme" ]] && ok "drop removes from order" || bad "drop failed, order=[$kept]" + +# all entries dropped -> RESOLVED_ORDER must be truly empty (not one "" element) +OV_DROP=([dropme]=1 [keepme]=1) +RESOLVED_ORDER=("$T/cat/dropme" "$T/cat/keepme") +apply_overrides_to_order +[[ ${#RESOLVED_ORDER[@]} -eq 0 ]] && ok "all-dropped -> empty order" || bad "all-dropped left ${#RESOLVED_ORDER[@]} elem(s): [${RESOLVED_ORDER[*]}]" +OV_DROP=() +rm -f "$OV" + +# overrides only apply on current; on 15.0 they are a no-op +VERSION_ID="15.0" +TB_OVERRIDES="$OV" # file gone; must not error, must clear maps +load_overrides +[[ ${#OV_DROP[@]} -eq 0 ]] && ok "overrides inert on 15.0" || bad "overrides applied on 15.0" +VERSION_ID="current" + +# --- cache ------------------------------------------------------------------ +PKG_CACHE=$(mktemp -d) +CACHE_ROOT="$PKG_CACHE/sha256-deadbeef" # simulate a resolved digest namespace +mkc() { mkdir -p "$CACHE_ROOT/$1/$2"; : > "$CACHE_ROOT/$1/$2/$3"; } + +mkc net libfoo "libfoo-1.1-x86_64-1_danix.txz" +[[ "$(cache_decision net libfoo 1.1)" == "cached" ]] && ok "cache hit on version match" || bad "cache_decision got [$(cache_decision net libfoo 1.1)]" +[[ "$(cache_decision net libfoo 1.2)" == "bump:1.1:1.2" ]] && ok "cache bump reported" || bad "cache bump got [$(cache_decision net libfoo 1.2)]" +[[ "$(cache_decision net libbar 1.0)" == "new" ]] && ok "cache new for absent" || bad "cache new got [$(cache_decision net libbar 1.0)]" + +hit="$(cache_path net libfoo 1.1)" +[[ "$hit" == "$CACHE_ROOT/net/libfoo/libfoo-1.1-x86_64-1_danix.txz" ]] && ok "cache_path returns hit" || bad "cache_path got [$hit]" +[[ -z "$(cache_path net libfoo 9.9)" ]] && ok "cache_path empty on miss" || bad "cache_path not empty on miss" + +srctmp=$(mktemp -d); : > "$srctmp/libfoo-1.2-x86_64-1_danix.txz" +cache_store net libfoo "$srctmp/libfoo-1.2-x86_64-1_danix.txz" +count=$(find "$CACHE_ROOT/net/libfoo" -name '*.t?z' | wc -l) +[[ "$count" -eq 1 ]] && ok "cache_store evicts to one file" || bad "cache_store left $count files" +[[ -e "$CACHE_ROOT/net/libfoo/libfoo-1.2-x86_64-1_danix.txz" ]] && ok "cache_store stored new file" || bad "cache_store did not store" +rm -rf "$srctmp" + +# disabled cache -> new +CACHE_ROOT_SAVE="$CACHE_ROOT"; USE_CACHE=0 +[[ "$(cache_decision net libfoo 1.2)" == "new" ]] && ok "--no-cache disables (new)" || bad "disabled cache got [$(cache_decision net libfoo 1.2)]" +USE_CACHE=1; CACHE_ROOT="$CACHE_ROOT_SAVE" + +# resolve_cache_root prunes stale digests of the SAME variant, keeps the other. +# No docker here, so the digest falls back to the tag-derived name; seed sibling +# dirs and confirm only the active-variant orphan is removed. +PRUNE_CACHE=$(mktemp -d) +PKG_CACHE="$PRUNE_CACHE"; ACTIVE_IMAGE="sbo-testbuild:current"; VERSION_ID="current" +mkdir -p "$PRUNE_CACHE/current-oldstale" "$PRUNE_CACHE/15.0-keepme" +resolve_cache_root +[[ -d "$CACHE_ROOT" ]] && ok "cache root created" || bad "cache root missing" +[[ ! -e "$PRUNE_CACHE/current-oldstale" ]] && ok "prune drops stale same-variant digest" || bad "stale current digest survived" +[[ -d "$PRUNE_CACHE/15.0-keepme" ]] && ok "prune keeps other-variant cache" || bad "15.0 cache wrongly pruned" +rm -rf "$PRUNE_CACHE" +VERSION_ID="current"; CACHE_ROOT="$CACHE_ROOT_SAVE" + +# --- BLOCKED-BY-DEP (depends_on_failed) ------------------------------------- +dead=(b) +if depends_on_failed "$T/cat/g" dead; then ok "g blocked when b dead"; else bad "g should block on b"; fi +if depends_on_failed "$T/cat/a" dead; then ok "a blocked when b dead (direct)"; else bad "a should block on b"; fi + +dead=(c) +if depends_on_failed "$T/cat/a" dead; then bad "a wrongly blocked on c"; else ok "a not directly blocked by c"; fi +if depends_on_failed "$T/cat/b" dead; then ok "b blocked when c dead"; else bad "b should block on c"; fi + +dead=() +if depends_on_failed "$T/cat/a" dead; then bad "a blocked with empty dead"; else ok "no block when nothing dead"; fi + +dead=("%README%") +if depends_on_failed "$T/cat/a" dead; then bad "%README% treated as dep"; else ok "%README% not treated as dep"; fi + +echo +echo "$pass passed, $fail failed" +[[ $fail -eq 0 ]] || exit 1 +echo "ALL LOGIC CHECKS PASS" |
