diff options
Diffstat (limited to '.extras/docs/specs')
| -rw-r--r-- | .extras/docs/specs/2026-07-13-docker-test-build-design.md | 258 | ||||
| -rw-r--r-- | .extras/docs/specs/2026-07-13-image-builder-design.md | 216 |
2 files changed, 0 insertions, 474 deletions
diff --git a/.extras/docs/specs/2026-07-13-docker-test-build-design.md b/.extras/docs/specs/2026-07-13-docker-test-build-design.md deleted file mode 100644 index 9f14d1e..0000000 --- a/.extras/docs/specs/2026-07-13-docker-test-build-design.md +++ /dev/null @@ -1,258 +0,0 @@ -# Docker test-build script - design - -Date: 2026-07-13 - -## Purpose - -A single bash script that verifies an already-published SBo package still -builds cleanly on a target Slackware version, inside a throwaway docker -container. This is the "test-build" step of this repo's maintenance loop -(track upstream -> bump -> **test-build** -> make tarball). - -The container is disposable, so unlike a real `slackrepo build` this runs -directly (no host install), and Claude may run it (per the repo's -`Who builds SlackBuilds` exception). - -## Scope and non-goals - -- One target package per run: `test-build <pkg>`. -- Two Slackware targets: `-current` (default) and `15.0` (`--stable`). -- Resolves the target's SBo dependency tree from the LOCAL SBo tree, builds - deps then target in the container, reports, discards everything. -- NOT a redistributable-package builder. The container's built `.txz` is - throwaway; the SBo submission tarball is made separately (post-commit hook) - only after a green run. -- Does NOT wrap slackrepo or sbo-batch-test. It REUSES sbo-batch-test's - resolver + build logic, adapted, but is an independent script. -- Shares the host kernel (docker). No kernel-module build claims. - -## Why not the overlay approach (sbo-batch-test) - -`sbo-batch-test` already resolves deps + builds + reports, but against a -LOCAL overlayfs over a 15.0 base. In docker the container IS the disposable -base, and overlayfs-over-docker-storage (itself overlayfs) is fragile. So we -drop the overlay layer and build directly in the container. We port -sbo-batch-test's dep resolver and its `build_one` chroot heredoc -(download -> md5 -> build -> installpkg -> status token) into a `docker run`. - -## Architecture - -Single self-contained bash script (`test-build`, location TBD in -implementation - likely `.extras/`). Host does resolution + orchestration; -the container does the actual building. - -``` -host: parse args, load config -host: pick version -> image tag + SBo tree -host: resolve dep tree (local tree, topo sort) [ported from sbo-batch-test] -host: apply current-vs-stable overrides -host: print final build order, Y/n confirm (--yes: still prints first) - docker run --rm -v <tree>:ro -v <pkg>:ro -v <cache/digest>:ro <image>: - for each dep in order: - cache hit (version match) -> installpkg from cache (CACHED) - else -> download/md5/build/installpkg + cache - target: download/md5/build/installpkg (always fresh) - target: sbopkglint on the built .txz -host: collect per-package results, print color summary -container discarded (--rm) -``` - -### Images (external input, not built by this script) - -The script does NOT build images. It consumes a ready image per version, -tagged locally: - -- current: `sbo-testbuild:current` -- 15.0: `sbo-testbuild:15.0` - -Each image is a FULL, patched Slackware install (SBo tests against a full -install; a minimal base causes false "missing dependency" results, a lesson -from sbo-batch-tester) plus `sbo-maintainer-tools` (for `sbopkglint`) and -`sbopkg` configured to that version's SBo repo (`:current` -> -current repo, -`:15.0` -> 15.0 repo). - -**How the images are produced is out of scope for this script.** A separate -nightly job (a LAN machine, images stored on the NAS, later served from a -local registry) builds and refreshes them. That builder is its own tool, -specced separately. - -For now the script assumes the tagged image is present locally: it runs -`docker run` against the tag and errors with a clear message if the image is -missing (pointing at the image-builder job). When the local registry exists, -a `docker pull` of the tag slots in ahead of the run via the same config, -no other change. - -Per-run cost is just the SlackBuild's own build time + ~1s container start. - -### Config (external file) - -`~/.config/sbo-testbuild/config`, empty defaults in-script, sourced if -present (same pattern as sbo-batch-test). Keys: - -```sh -SBO_TREE_CURRENT=/path/to/SBo-current # local SBo tree, -current -SBO_TREE_STABLE=/path/to/SBo-15.0 # local SBo tree, 15.0 -IMAGE_CURRENT=sbo-testbuild:current # ready image tag (built elsewhere) -IMAGE_STABLE=sbo-testbuild:15.0 -LOG_ROOT=/path/to/logs -PKG_CACHE=/path/to/cache # dep cache; empty = disabled -``` - -The version flag selects BOTH the image and the tree together (default -current; `--stable` / `15.0` switches both), so tree and image never mismatch. - -### current-vs-stable overrides - -A data file next to the config, `~/.config/sbo-testbuild/overrides`. It -encodes the known deltas between building on 15.0 and on -current. Applied -ONLY when the target version is `-current` (15.0 is the SBo baseline). Three -rule kinds: - -``` -# dep present in 15.0 but already in -current base -> drop from order -drop: rust -# dep renamed on -current -> map old -> new before lookup -rename: python3-foo -> foo -# dep removed from the -current tree -> fetch from SBo instead of local tree -fetch: somelib -``` - -Format: one rule per line, `kind: value` (rename uses `old -> new`), `#` -comments and blank lines ignored. - -### Dependency resolution + unknown deps - -Resolver is ported from sbo-batch-test (`resolve_target` / `_resolve_visit`, -DFS topo sort + cycle detection, `installed_in_base` check). Runs on the host -against the selected tree. - -After resolving and applying overrides, every dep must be accounted for: -in base, in the tree, or covered by a `fetch:` rule. If a dep is none of -those (and no rule covers it), the script **stops before building**, reports -it as UNMET, and asks the user to add an override rule and rerun, or abort. -Never silently skip or guess. - -### Confirmation - -Always print the final build order before building, with overrides marked -(dropped / renamed / fetch). Then `Y/n` to proceed. `--yes` skips the prompt -for non-interactive/agent runs but STILL prints the order first. - -`--dry-run` resolves, applies overrides, prints the order, and exits without -building (no confirm). - -### Build in container (ported from build_one) - -Deps then target, in order. For each package, inside the container: - -1. download sources (arch-specific `DOWNLOAD_x86_64`/`MD5SUM_x86_64` when - present, else `DOWNLOAD`/`MD5SUM`) -2. verify md5 -3. `bash <pkg>.SlackBuild` with `OUTPUT` forced to a known dir -4. `installpkg` the resulting `.txz` -5. write a status token read back by the host - -`fetch:` deps (removed from the -current tree) are pulled via the baked-in -`sbopkg`, which points at the image's matching SBo repo; everything else -builds from the mounted local tree. Start light: handle only the few known -`fetch:` cases, add rules as real packages need them. - -### Dependency cache - -Ported from sbo-batch-tester (`cache_decision`/`cache_path`/`cache_store`/ -`version_of`). A host dir caches built dep packages so consecutive runs reuse -them instead of rebuilding the whole tree. - -- Config `PKG_CACHE=/path` enables it; empty (default) disables. `--no-cache` - forces a full rebuild for one run. -- Bind-mounted read-only into the container; a cached dep is `installpkg`ed - instead of built (status `CACHED`). Storing is host-side: a freshly built - package is written to `OUTPUT` (a mounted dir the host reads back), and the - host copies it into the cache after the build (`cache_store`), so the cache - mount stays read-only. The **target always builds fresh** and refreshes its - own cache entry; it is never `CACHED`. -- Key = prog + version (build/arch/tag ignored). Layout mirrors the SBo tree: - `$PKG_CACHE/<image-digest>/<category>/<prog>/<prog>-<ver>-...txz`, one .txz - per prog. -- **Invalidation by image digest.** The cache is namespaced by the running - image's digest. When the digest changes (image updated by the nightly job), - the old namespace is stale: its deps were built against the previous image, - so they are ignored and rebuilt once under the new digest. This also keeps - the `:current` and `:15.0` caches separate (different images, different - digests). Old digest namespaces can be pruned. -- A per-dep version bump (`bump: OLD -> NEW`) rebuilds and re-caches that dep. -- Build-order line shows the outcome per dep: `cached (1.1)`, - `rebuild: 1.0 -> 1.1`, or `build (new)`; `--dry-run` shows the same without - building. - -### Target lint - -After the target builds, run `sbopkglint` on its `.txz` (tool baked into the -image). Fail-soft: findings reported but do not change SUCCESS; missing tool -prints a skip note. Target only (deps are vetted). - -### Report - -Per-package status + a color summary (screen) and a plain-text log under -`LOG_ROOT/<timestamp>/`: - -``` -<timestamp>/ - <prog>.log per-package full build/install output - summary.log plain recap - build-order.txt the resolved+overridden order actually used -``` - -Status values (from sbo-batch-test): -`SUCCESS CACHED DOWNLOAD-FAILED MD5-MISMATCH BUILD-FAILED INSTALL-FAILED -BLOCKED-BY-DEP UNMET-DEP`. `CACHED` = a dep installed from the cache instead -of rebuilt (target is never CACHED). `%README%` deps flagged as a reminder, -not built. - -A fully green run tells the user it is safe to make the SBo submission tarball -on the host (the container is no longer needed). - -## Options (planned) - -| Option | Effect | -|--------|--------| -| `--stable` / `15.0` | Target 15.0 (image + tree). Default is -current. | -| `--dry-run` | Resolve + apply overrides + print order, no build. | -| `--yes` | Skip the Y/n confirm (still prints the order). | -| `--no-cache` | Rebuild all deps this run, ignore/refresh the cache. | -| `--no-color` | Disable ANSI (auto-off when not a TTY). | -| `-h`, `--help` | Usage. | - -## Self-check - -A `test-logic.sh` (host, no docker) covering the pure logic: - -- dep topo order + cycle detection (ported tests from sbo-batch-test) -- override application: drop removes, rename maps, fetch marks source -- unknown-dep -> UNMET stop -- BLOCKED-BY-DEP propagation -- cache decision: `cached`/`rebuild: OLD -> NEW`/`build (new)`, digest - namespacing, version-bump eviction (ported from sbo-batch-tester's tests) - -Docker/build/installpkg paths are out of the self-check's reach (verified by -running a real target), same boundary as sbo-batch-test. - -## Resolved decisions - -- **Image**: a ready FULL-install image per version (full pkg set + - `sbo-maintainer-tools` + `sbopkg` on that version's repo), built by a - separate nightly job, NOT by this script. Script consumes it by tag. -- **Missing image**: error clearly, point at the image-builder job. A local - registry `docker pull` slots in later via the same tag, no script change. -- **`fetch:`**: via the image's `sbopkg` (per-version repo). Start with only - the few known cases; grow the override rules as needed. -- **Script location**: `.extras/` (a tool, not a package). - -## Out of scope / later - -- The nightly image-builder (full-install + tools + sbopkg per version, NAS - storage, local registry). Its own tool, specced when the LAN machine is set - up. -- Local registry pull step in this script (slots into the same config tag). -``` - diff --git a/.extras/docs/specs/2026-07-13-image-builder-design.md b/.extras/docs/specs/2026-07-13-image-builder-design.md deleted file mode 100644 index 9ee1a11..0000000 --- a/.extras/docs/specs/2026-07-13-image-builder-design.md +++ /dev/null @@ -1,216 +0,0 @@ -# sbo-testbuild image builder — design - -Date: 2026-07-13 -Status: approved (design) - -## Purpose - -Produce and serve the docker images the `.extras/test-build` tool consumes. -`.extras/test-build` verifies a published SBo package still builds on a target -Slackware version inside a throwaway container; it does **not** build the -images, it only checks image presence by tag and errors if absent. This is the -missing image-builder job. - -Images produced (x86_64 only for now): - -- `sbo-testbuild:current` — Slackware -current, full install + sbopkg + - sbo-maintainer-tools -- `sbo-testbuild:15.0` — Slackware 15.0, same - -Served from a LAN docker registry so `.extras/test-build`'s `require_image` -can `docker pull` them, and so `installed_in_base` can later query the image's -package db. - -## Infrastructure - -Self-hosted on a Slackware VM (proxmox node), fully offline from the public -internet for the Slackware bits: - -- **Host**: `docker.noland.dnx` (LAN DNS), Slackware x86_64, 4 vCPU / 4 GB RAM - / 80 GB disk. -- **NAS trees**: two standard Slackware mirror trees NFS-mounted read-only: - - `/mnt/nas/slackware64-current` - - `/mnt/nas/slackware64-15.0` - Each is a full mirror tree (`PACKAGES.TXT`, `ChangeLog.txt`, `slackware64/` - series dirs, `patches/`, `extra/`). -- **Registry**: `registry:2` container on the VM, port 5000, plain HTTP (LAN - only). Pull path `docker.noland.dnx:5000/sbo-testbuild:{tag}`. -- **Local .txz drop**: `/opt/sbo-testbuild/pkgs/` holds `sbopkg-*.txz` and - `sbo-maintainer-tools-*.txz`, provided by the user (sbopkg packaged from the - user's SBo repo). Installed into the image, not built in it (hermetic, no - build-time deps). - -Both `docker.noland.dnx` and every client that pulls (this dev box, the -`buildsystem` VM) need the registry marked insecure in -`/etc/docker/daemon.json`: -`{"insecure-registries":["docker.noland.dnx:5000"]}`, then `docker` restarted. - -i586 is out of scope for now; the variant list is written to extend to it -later without restructuring. - -## Architecture - -Three scripts, chained, adapted from the forge repo -`slackware/docker-images` (`scripts/bootstrap.sh`, `build-full-image.sh`, -`build-builder-image.sh`). The forge `build-builder-image.sh` is **not** used: -the full image already ships the toolchain (series d, kde, x, etc.), so a -separate builder layer is redundant here. - -``` -bootstrap.sh FROM scratch; installpkg base pkgs read from NAS - → sbo-base:{ver} via file://; configure rootfs for container use. - -build-full-image.sh FROM sbo-base:{ver}; slackpkg install every series - → sbo-full:{ver} (a ap d e f k kde l n t tcl x xap xfce y). - -build-sbo-testbuild.sh FROM sbo-full:{ver}; installpkg sbopkg + - → sbo-testbuild:{ver} sbo-maintainer-tools from /opt/sbo-testbuild/pkgs. -``` - -All three push to `docker.noland.dnx:5000`. - -### Namespacing - -Forge scripts push to `registry.slackware.nl/slackware/slackware*`. Re-namespace -to the LAN registry with our own image names: - -- `docker.noland.dnx:5000/sbo-base:{ver}` -- `docker.noland.dnx:5000/sbo-full:{ver}` -- `docker.noland.dnx:5000/sbo-testbuild:{ver}` - -`sbo-base` and `sbo-full` are intermediate; `.extras/test-build` only pulls -`sbo-testbuild`. - -## Configuration - -Single shared file `config`, sourced by all scripts and the test: - -```sh -REGISTRY=docker.noland.dnx:5000 -MIRROR=file:///mnt/nas -VARIANTS=(current 15.0) # x86_64 only for now -PKGDIR=/opt/sbo-testbuild/pkgs # sbopkg + sbo-maintainer-tools .txz -``` - -The forge scripts hardcode `MIRROR=https://slackware.nl/slackware` and -`REGISTRY=registry.slackware.nl/...`; both move to the shared config. - -### NAS path mapping - -bootstrap derives the per-variant repo dir as `slackware64-${VERSION}` for -x86_64 (`DIRSUFFIX=64`). With `MIRROR=file:///mnt/nas` this resolves to -`file:///mnt/nas/slackware64-current/...` and `.../slackware64-15.0/...`, -matching the mount points. Mounts must be named to match. - -## Patches to the forge scripts - -1. **`file://`-aware fetch.** Forge downloads `PACKAGES.TXT` and packages with - `curl` over http. Replace with one helper in `lib.sh` that detects a - `file://` (or bare-local) MIRROR and uses `cp`, otherwise `curl`. curl's - own `file://` support is unreliable across builds, so branch explicitly. -2. **Drop i586.** `ALL_VARIANTS` becomes the config `VARIANTS` (x86_64 only). - All the `ARCH_SUFFIX`/`-i586` machinery collapses to the x86_64 path. -3. **Re-namespace + LAN registry** (see Namespacing). -4. **MIRROR → `file:///mnt/nas`** (see config). - -## Data flow and rebuild gating - -Nightly cron on the VM (root; `bootstrap` needs `installpkg`): - -```cron - 0 4 * * * bootstrap.sh -15 4 * * * build-full-image.sh -30 4 * * * build-sbo-testbuild.sh -``` - -Rebuild gating (kept from the forge scripts, extended): - -- **bootstrap**: sha256 of the NAS `ChangeLog.txt` (read via `file://`) - compared to a stored hash under `HASH_DIR`; skip the variant if unchanged. -- **build-full-image**: base image digest recorded as a label on the full - image; pull base, compare, skip if unchanged. -- **build-sbo-testbuild**: full image digest label, same pattern; **plus** a - hash of the `.txz` set in `PKGDIR`, so a sbopkg / sbo-maintainer-tools bump - forces a rebuild even when the full image is unchanged. - -Net effect: a quiet NAS tree makes the whole chain no-op in seconds; a --current churn cascades base → full → testbuild. - -### Local-registry digest consistency - -The forge scripts read the base/full digest via `docker inspect RepoDigests` -after `docker pull`. Self-hosted, those intermediate images live in the LAN -registry, so pull-then-inspect must target `docker.noland.dnx:5000`. Since -every image name is derived from `REGISTRY`, re-namespacing handles this with -no extra logic. - -## Error handling - -- `set -euo pipefail` in every script (forge already has it). -- **Missing NAS mount**: guard `[[ -d /mnt/nas/slackware64-$ver ]]` first; - `_err` if absent. -- **Missing `.txz`**: `build-sbo-testbuild` errors before building if - `PKGDIR` lacks a `sbopkg-*.txz` or `sbo-maintainer-tools-*.txz` — never push - a testbuild image missing sbopkg. -- **Registry unreachable**: `docker push` fails under `set -e`; the chain - aborts. The base/full may already be built locally; cron retries next night. - Acceptable. -- **Per-variant isolation**: one variant failing (e.g. current) must not block - the other (15.0). Each variant runs wrapped; log the failure, continue, and - exit non-zero at the end so cron logs surface it. - -## Testing - -Pure-logic pieces get a `test-image-builder.sh` self-check, matching the -existing `.extras/test-build` pattern (`.extras/test-logic.sh`, assert-based, -no framework). No docker daemon in tests — hermetic. - -- **`file://` fetch helper**: local file resolves via cp; missing local file - errors; an http MIRROR still routes to curl. -- **`.txz`-set hash**: stable across file reordering; changes when a file's - content changes. -- **variant guard**: a missing mount dir yields non-zero. - -The real end-to-end chain is verified by a manual first run on the VM (the user -runs it and reports back), the same division of labor as `.extras/test-build` -Task 10. - -## Layout - -``` -.extras/image-builder/ -├── config REGISTRY, MIRROR, VARIANTS, PKGDIR -├── lib.sh shared: _log/_warn/_err, fetch, txz_hash, guards -├── bootstrap.sh base image (root; installpkg from NAS) -├── build-full-image.sh full image (slackpkg all series) -├── build-sbo-testbuild.sh testbuild image (installpkg sbopkg + tools) -├── test-image-builder.sh pure-logic self-check -└── README VM-side setup, reproducible -``` - -Lives in `.extras/` for now (per the user), relocatable later. - -The README carries the user-side checklist so it is reproducible and not -trapped in chat: - -- VM specs (4/4/80), Slackware x86_64, docker installed. -- Two NFS mounts (ro), root-readable, named `slackware64-current` / - `slackware64-15.0` under `/mnt/nas`. -- `registry:2` container: `-v /opt/registry/data:/var/lib/registry`, - `--restart=always`, port 5000. -- `/etc/docker/daemon.json` insecure-registry entry on the VM **and** on every - pulling client (this dev box, `buildsystem`), docker restarted. -- Drop `sbopkg-*.txz` and `sbo-maintainer-tools-*.txz` into - `/opt/sbo-testbuild/pkgs/`. -- Install the three cron lines. -- VM static IP / LAN DNS for `docker.noland.dnx`. - -## Out of scope - -- i586 images (extensible later via `VARIANTS` / arch machinery). -- Building sbopkg / sbo-maintainer-tools from source in the image (installed - from prebuilt `.txz`). -- The forge `build-builder-image.sh` toolchain layer (redundant with full). -- `installed_in_base` implementation in `.extras/test-build` — unblocked by - this work (queries the built image's package db) but tracked separately. -- Registry auth / TLS (LAN-only plain HTTP). |