summaryrefslogtreecommitdiffstats
path: root/docs/specs
diff options
context:
space:
mode:
authorDanilo M. <danix@danix.xyz>2026-07-14 16:59:44 +0200
committerDanilo M. <danix@danix.xyz>2026-07-14 16:59:44 +0200
commit3cd610c212076d438f2d1103e0fc584a310011d5 (patch)
treeb82bd71ff139408dc84f685345862f944653966b /docs/specs
downloadsbo-dockerbuild-3cd610c212076d438f2d1103e0fc584a310011d5.tar.gz
sbo-dockerbuild-3cd610c212076d438f2d1103e0fc584a310011d5.zip
initial commit: docker test-build toolchain
Extracted from the .extras/ dir of the sbo-slackbuilds package repo, where it grew from a throwaway helper into a standalone tool. Git history starts fresh; the original 41-commit development log is preserved in HISTORY.md. Contents: the test-build CLI (resolves a local SlackBuild's deps from a configured SBo tree and builds it in a throwaway container, lints, caches deps), the image-builder chain that produces the images it consumes, their pure-logic self-checks, design specs/plans, an install.sh for ~/bin, and docs. Licensed GPLv2-only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Diffstat (limited to 'docs/specs')
-rw-r--r--docs/specs/2026-07-13-docker-test-build-design.md258
-rw-r--r--docs/specs/2026-07-13-image-builder-design.md216
2 files changed, 474 insertions, 0 deletions
diff --git a/docs/specs/2026-07-13-docker-test-build-design.md b/docs/specs/2026-07-13-docker-test-build-design.md
new file mode 100644
index 0000000..9f14d1e
--- /dev/null
+++ b/docs/specs/2026-07-13-docker-test-build-design.md
@@ -0,0 +1,258 @@
+# Docker test-build script - design
+
+Date: 2026-07-13
+
+## Purpose
+
+A single bash script that verifies an already-published SBo package still
+builds cleanly on a target Slackware version, inside a throwaway docker
+container. This is the "test-build" step of this repo's maintenance loop
+(track upstream -> bump -> **test-build** -> make tarball).
+
+The container is disposable, so unlike a real `slackrepo build` this runs
+directly (no host install), and Claude may run it (per the repo's
+`Who builds SlackBuilds` exception).
+
+## Scope and non-goals
+
+- One target package per run: `test-build <pkg>`.
+- Two Slackware targets: `-current` (default) and `15.0` (`--stable`).
+- Resolves the target's SBo dependency tree from the LOCAL SBo tree, builds
+ deps then target in the container, reports, discards everything.
+- NOT a redistributable-package builder. The container's built `.txz` is
+ throwaway; the SBo submission tarball is made separately (post-commit hook)
+ only after a green run.
+- Does NOT wrap slackrepo or sbo-batch-test. It REUSES sbo-batch-test's
+ resolver + build logic, adapted, but is an independent script.
+- Shares the host kernel (docker). No kernel-module build claims.
+
+## Why not the overlay approach (sbo-batch-test)
+
+`sbo-batch-test` already resolves deps + builds + reports, but against a
+LOCAL overlayfs over a 15.0 base. In docker the container IS the disposable
+base, and overlayfs-over-docker-storage (itself overlayfs) is fragile. So we
+drop the overlay layer and build directly in the container. We port
+sbo-batch-test's dep resolver and its `build_one` chroot heredoc
+(download -> md5 -> build -> installpkg -> status token) into a `docker run`.
+
+## Architecture
+
+Single self-contained bash script (`test-build`, location TBD in
+implementation - likely `.extras/`). Host does resolution + orchestration;
+the container does the actual building.
+
+```
+host: parse args, load config
+host: pick version -> image tag + SBo tree
+host: resolve dep tree (local tree, topo sort) [ported from sbo-batch-test]
+host: apply current-vs-stable overrides
+host: print final build order, Y/n confirm (--yes: still prints first)
+ docker run --rm -v <tree>:ro -v <pkg>:ro -v <cache/digest>:ro <image>:
+ for each dep in order:
+ cache hit (version match) -> installpkg from cache (CACHED)
+ else -> download/md5/build/installpkg + cache
+ target: download/md5/build/installpkg (always fresh)
+ target: sbopkglint on the built .txz
+host: collect per-package results, print color summary
+container discarded (--rm)
+```
+
+### Images (external input, not built by this script)
+
+The script does NOT build images. It consumes a ready image per version,
+tagged locally:
+
+- current: `sbo-testbuild:current`
+- 15.0: `sbo-testbuild:15.0`
+
+Each image is a FULL, patched Slackware install (SBo tests against a full
+install; a minimal base causes false "missing dependency" results, a lesson
+from sbo-batch-tester) plus `sbo-maintainer-tools` (for `sbopkglint`) and
+`sbopkg` configured to that version's SBo repo (`:current` -> -current repo,
+`:15.0` -> 15.0 repo).
+
+**How the images are produced is out of scope for this script.** A separate
+nightly job (a LAN machine, images stored on the NAS, later served from a
+local registry) builds and refreshes them. That builder is its own tool,
+specced separately.
+
+For now the script assumes the tagged image is present locally: it runs
+`docker run` against the tag and errors with a clear message if the image is
+missing (pointing at the image-builder job). When the local registry exists,
+a `docker pull` of the tag slots in ahead of the run via the same config,
+no other change.
+
+Per-run cost is just the SlackBuild's own build time + ~1s container start.
+
+### Config (external file)
+
+`~/.config/sbo-testbuild/config`, empty defaults in-script, sourced if
+present (same pattern as sbo-batch-test). Keys:
+
+```sh
+SBO_TREE_CURRENT=/path/to/SBo-current # local SBo tree, -current
+SBO_TREE_STABLE=/path/to/SBo-15.0 # local SBo tree, 15.0
+IMAGE_CURRENT=sbo-testbuild:current # ready image tag (built elsewhere)
+IMAGE_STABLE=sbo-testbuild:15.0
+LOG_ROOT=/path/to/logs
+PKG_CACHE=/path/to/cache # dep cache; empty = disabled
+```
+
+The version flag selects BOTH the image and the tree together (default
+current; `--stable` / `15.0` switches both), so tree and image never mismatch.
+
+### current-vs-stable overrides
+
+A data file next to the config, `~/.config/sbo-testbuild/overrides`. It
+encodes the known deltas between building on 15.0 and on -current. Applied
+ONLY when the target version is `-current` (15.0 is the SBo baseline). Three
+rule kinds:
+
+```
+# dep present in 15.0 but already in -current base -> drop from order
+drop: rust
+# dep renamed on -current -> map old -> new before lookup
+rename: python3-foo -> foo
+# dep removed from the -current tree -> fetch from SBo instead of local tree
+fetch: somelib
+```
+
+Format: one rule per line, `kind: value` (rename uses `old -> new`), `#`
+comments and blank lines ignored.
+
+### Dependency resolution + unknown deps
+
+Resolver is ported from sbo-batch-test (`resolve_target` / `_resolve_visit`,
+DFS topo sort + cycle detection, `installed_in_base` check). Runs on the host
+against the selected tree.
+
+After resolving and applying overrides, every dep must be accounted for:
+in base, in the tree, or covered by a `fetch:` rule. If a dep is none of
+those (and no rule covers it), the script **stops before building**, reports
+it as UNMET, and asks the user to add an override rule and rerun, or abort.
+Never silently skip or guess.
+
+### Confirmation
+
+Always print the final build order before building, with overrides marked
+(dropped / renamed / fetch). Then `Y/n` to proceed. `--yes` skips the prompt
+for non-interactive/agent runs but STILL prints the order first.
+
+`--dry-run` resolves, applies overrides, prints the order, and exits without
+building (no confirm).
+
+### Build in container (ported from build_one)
+
+Deps then target, in order. For each package, inside the container:
+
+1. download sources (arch-specific `DOWNLOAD_x86_64`/`MD5SUM_x86_64` when
+ present, else `DOWNLOAD`/`MD5SUM`)
+2. verify md5
+3. `bash <pkg>.SlackBuild` with `OUTPUT` forced to a known dir
+4. `installpkg` the resulting `.txz`
+5. write a status token read back by the host
+
+`fetch:` deps (removed from the -current tree) are pulled via the baked-in
+`sbopkg`, which points at the image's matching SBo repo; everything else
+builds from the mounted local tree. Start light: handle only the few known
+`fetch:` cases, add rules as real packages need them.
+
+### Dependency cache
+
+Ported from sbo-batch-tester (`cache_decision`/`cache_path`/`cache_store`/
+`version_of`). A host dir caches built dep packages so consecutive runs reuse
+them instead of rebuilding the whole tree.
+
+- Config `PKG_CACHE=/path` enables it; empty (default) disables. `--no-cache`
+ forces a full rebuild for one run.
+- Bind-mounted read-only into the container; a cached dep is `installpkg`ed
+ instead of built (status `CACHED`). Storing is host-side: a freshly built
+ package is written to `OUTPUT` (a mounted dir the host reads back), and the
+ host copies it into the cache after the build (`cache_store`), so the cache
+ mount stays read-only. The **target always builds fresh** and refreshes its
+ own cache entry; it is never `CACHED`.
+- Key = prog + version (build/arch/tag ignored). Layout mirrors the SBo tree:
+ `$PKG_CACHE/<image-digest>/<category>/<prog>/<prog>-<ver>-...txz`, one .txz
+ per prog.
+- **Invalidation by image digest.** The cache is namespaced by the running
+ image's digest. When the digest changes (image updated by the nightly job),
+ the old namespace is stale: its deps were built against the previous image,
+ so they are ignored and rebuilt once under the new digest. This also keeps
+ the `:current` and `:15.0` caches separate (different images, different
+ digests). Old digest namespaces can be pruned.
+- A per-dep version bump (`bump: OLD -> NEW`) rebuilds and re-caches that dep.
+- Build-order line shows the outcome per dep: `cached (1.1)`,
+ `rebuild: 1.0 -> 1.1`, or `build (new)`; `--dry-run` shows the same without
+ building.
+
+### Target lint
+
+After the target builds, run `sbopkglint` on its `.txz` (tool baked into the
+image). Fail-soft: findings reported but do not change SUCCESS; missing tool
+prints a skip note. Target only (deps are vetted).
+
+### Report
+
+Per-package status + a color summary (screen) and a plain-text log under
+`LOG_ROOT/<timestamp>/`:
+
+```
+<timestamp>/
+ <prog>.log per-package full build/install output
+ summary.log plain recap
+ build-order.txt the resolved+overridden order actually used
+```
+
+Status values (from sbo-batch-test):
+`SUCCESS CACHED DOWNLOAD-FAILED MD5-MISMATCH BUILD-FAILED INSTALL-FAILED
+BLOCKED-BY-DEP UNMET-DEP`. `CACHED` = a dep installed from the cache instead
+of rebuilt (target is never CACHED). `%README%` deps flagged as a reminder,
+not built.
+
+A fully green run tells the user it is safe to make the SBo submission tarball
+on the host (the container is no longer needed).
+
+## Options (planned)
+
+| Option | Effect |
+|--------|--------|
+| `--stable` / `15.0` | Target 15.0 (image + tree). Default is -current. |
+| `--dry-run` | Resolve + apply overrides + print order, no build. |
+| `--yes` | Skip the Y/n confirm (still prints the order). |
+| `--no-cache` | Rebuild all deps this run, ignore/refresh the cache. |
+| `--no-color` | Disable ANSI (auto-off when not a TTY). |
+| `-h`, `--help` | Usage. |
+
+## Self-check
+
+A `test-logic.sh` (host, no docker) covering the pure logic:
+
+- dep topo order + cycle detection (ported tests from sbo-batch-test)
+- override application: drop removes, rename maps, fetch marks source
+- unknown-dep -> UNMET stop
+- BLOCKED-BY-DEP propagation
+- cache decision: `cached`/`rebuild: OLD -> NEW`/`build (new)`, digest
+ namespacing, version-bump eviction (ported from sbo-batch-tester's tests)
+
+Docker/build/installpkg paths are out of the self-check's reach (verified by
+running a real target), same boundary as sbo-batch-test.
+
+## Resolved decisions
+
+- **Image**: a ready FULL-install image per version (full pkg set +
+ `sbo-maintainer-tools` + `sbopkg` on that version's repo), built by a
+ separate nightly job, NOT by this script. Script consumes it by tag.
+- **Missing image**: error clearly, point at the image-builder job. A local
+ registry `docker pull` slots in later via the same tag, no script change.
+- **`fetch:`**: via the image's `sbopkg` (per-version repo). Start with only
+ the few known cases; grow the override rules as needed.
+- **Script location**: `.extras/` (a tool, not a package).
+
+## Out of scope / later
+
+- The nightly image-builder (full-install + tools + sbopkg per version, NAS
+ storage, local registry). Its own tool, specced when the LAN machine is set
+ up.
+- Local registry pull step in this script (slots into the same config tag).
+```
+
diff --git a/docs/specs/2026-07-13-image-builder-design.md b/docs/specs/2026-07-13-image-builder-design.md
new file mode 100644
index 0000000..9ee1a11
--- /dev/null
+++ b/docs/specs/2026-07-13-image-builder-design.md
@@ -0,0 +1,216 @@
+# sbo-testbuild image builder — design
+
+Date: 2026-07-13
+Status: approved (design)
+
+## Purpose
+
+Produce and serve the docker images the `.extras/test-build` tool consumes.
+`.extras/test-build` verifies a published SBo package still builds on a target
+Slackware version inside a throwaway container; it does **not** build the
+images, it only checks image presence by tag and errors if absent. This is the
+missing image-builder job.
+
+Images produced (x86_64 only for now):
+
+- `sbo-testbuild:current` — Slackware -current, full install + sbopkg +
+ sbo-maintainer-tools
+- `sbo-testbuild:15.0` — Slackware 15.0, same
+
+Served from a LAN docker registry so `.extras/test-build`'s `require_image`
+can `docker pull` them, and so `installed_in_base` can later query the image's
+package db.
+
+## Infrastructure
+
+Self-hosted on a Slackware VM (proxmox node), fully offline from the public
+internet for the Slackware bits:
+
+- **Host**: `docker.noland.dnx` (LAN DNS), Slackware x86_64, 4 vCPU / 4 GB RAM
+ / 80 GB disk.
+- **NAS trees**: two standard Slackware mirror trees NFS-mounted read-only:
+ - `/mnt/nas/slackware64-current`
+ - `/mnt/nas/slackware64-15.0`
+ Each is a full mirror tree (`PACKAGES.TXT`, `ChangeLog.txt`, `slackware64/`
+ series dirs, `patches/`, `extra/`).
+- **Registry**: `registry:2` container on the VM, port 5000, plain HTTP (LAN
+ only). Pull path `docker.noland.dnx:5000/sbo-testbuild:{tag}`.
+- **Local .txz drop**: `/opt/sbo-testbuild/pkgs/` holds `sbopkg-*.txz` and
+ `sbo-maintainer-tools-*.txz`, provided by the user (sbopkg packaged from the
+ user's SBo repo). Installed into the image, not built in it (hermetic, no
+ build-time deps).
+
+Both `docker.noland.dnx` and every client that pulls (this dev box, the
+`buildsystem` VM) need the registry marked insecure in
+`/etc/docker/daemon.json`:
+`{"insecure-registries":["docker.noland.dnx:5000"]}`, then `docker` restarted.
+
+i586 is out of scope for now; the variant list is written to extend to it
+later without restructuring.
+
+## Architecture
+
+Three scripts, chained, adapted from the forge repo
+`slackware/docker-images` (`scripts/bootstrap.sh`, `build-full-image.sh`,
+`build-builder-image.sh`). The forge `build-builder-image.sh` is **not** used:
+the full image already ships the toolchain (series d, kde, x, etc.), so a
+separate builder layer is redundant here.
+
+```
+bootstrap.sh FROM scratch; installpkg base pkgs read from NAS
+ → sbo-base:{ver} via file://; configure rootfs for container use.
+
+build-full-image.sh FROM sbo-base:{ver}; slackpkg install every series
+ → sbo-full:{ver} (a ap d e f k kde l n t tcl x xap xfce y).
+
+build-sbo-testbuild.sh FROM sbo-full:{ver}; installpkg sbopkg +
+ → sbo-testbuild:{ver} sbo-maintainer-tools from /opt/sbo-testbuild/pkgs.
+```
+
+All three push to `docker.noland.dnx:5000`.
+
+### Namespacing
+
+Forge scripts push to `registry.slackware.nl/slackware/slackware*`. Re-namespace
+to the LAN registry with our own image names:
+
+- `docker.noland.dnx:5000/sbo-base:{ver}`
+- `docker.noland.dnx:5000/sbo-full:{ver}`
+- `docker.noland.dnx:5000/sbo-testbuild:{ver}`
+
+`sbo-base` and `sbo-full` are intermediate; `.extras/test-build` only pulls
+`sbo-testbuild`.
+
+## Configuration
+
+Single shared file `config`, sourced by all scripts and the test:
+
+```sh
+REGISTRY=docker.noland.dnx:5000
+MIRROR=file:///mnt/nas
+VARIANTS=(current 15.0) # x86_64 only for now
+PKGDIR=/opt/sbo-testbuild/pkgs # sbopkg + sbo-maintainer-tools .txz
+```
+
+The forge scripts hardcode `MIRROR=https://slackware.nl/slackware` and
+`REGISTRY=registry.slackware.nl/...`; both move to the shared config.
+
+### NAS path mapping
+
+bootstrap derives the per-variant repo dir as `slackware64-${VERSION}` for
+x86_64 (`DIRSUFFIX=64`). With `MIRROR=file:///mnt/nas` this resolves to
+`file:///mnt/nas/slackware64-current/...` and `.../slackware64-15.0/...`,
+matching the mount points. Mounts must be named to match.
+
+## Patches to the forge scripts
+
+1. **`file://`-aware fetch.** Forge downloads `PACKAGES.TXT` and packages with
+ `curl` over http. Replace with one helper in `lib.sh` that detects a
+ `file://` (or bare-local) MIRROR and uses `cp`, otherwise `curl`. curl's
+ own `file://` support is unreliable across builds, so branch explicitly.
+2. **Drop i586.** `ALL_VARIANTS` becomes the config `VARIANTS` (x86_64 only).
+ All the `ARCH_SUFFIX`/`-i586` machinery collapses to the x86_64 path.
+3. **Re-namespace + LAN registry** (see Namespacing).
+4. **MIRROR → `file:///mnt/nas`** (see config).
+
+## Data flow and rebuild gating
+
+Nightly cron on the VM (root; `bootstrap` needs `installpkg`):
+
+```cron
+ 0 4 * * * bootstrap.sh
+15 4 * * * build-full-image.sh
+30 4 * * * build-sbo-testbuild.sh
+```
+
+Rebuild gating (kept from the forge scripts, extended):
+
+- **bootstrap**: sha256 of the NAS `ChangeLog.txt` (read via `file://`)
+ compared to a stored hash under `HASH_DIR`; skip the variant if unchanged.
+- **build-full-image**: base image digest recorded as a label on the full
+ image; pull base, compare, skip if unchanged.
+- **build-sbo-testbuild**: full image digest label, same pattern; **plus** a
+ hash of the `.txz` set in `PKGDIR`, so a sbopkg / sbo-maintainer-tools bump
+ forces a rebuild even when the full image is unchanged.
+
+Net effect: a quiet NAS tree makes the whole chain no-op in seconds; a
+-current churn cascades base → full → testbuild.
+
+### Local-registry digest consistency
+
+The forge scripts read the base/full digest via `docker inspect RepoDigests`
+after `docker pull`. Self-hosted, those intermediate images live in the LAN
+registry, so pull-then-inspect must target `docker.noland.dnx:5000`. Since
+every image name is derived from `REGISTRY`, re-namespacing handles this with
+no extra logic.
+
+## Error handling
+
+- `set -euo pipefail` in every script (forge already has it).
+- **Missing NAS mount**: guard `[[ -d /mnt/nas/slackware64-$ver ]]` first;
+ `_err` if absent.
+- **Missing `.txz`**: `build-sbo-testbuild` errors before building if
+ `PKGDIR` lacks a `sbopkg-*.txz` or `sbo-maintainer-tools-*.txz` — never push
+ a testbuild image missing sbopkg.
+- **Registry unreachable**: `docker push` fails under `set -e`; the chain
+ aborts. The base/full may already be built locally; cron retries next night.
+ Acceptable.
+- **Per-variant isolation**: one variant failing (e.g. current) must not block
+ the other (15.0). Each variant runs wrapped; log the failure, continue, and
+ exit non-zero at the end so cron logs surface it.
+
+## Testing
+
+Pure-logic pieces get a `test-image-builder.sh` self-check, matching the
+existing `.extras/test-build` pattern (`.extras/test-logic.sh`, assert-based,
+no framework). No docker daemon in tests — hermetic.
+
+- **`file://` fetch helper**: local file resolves via cp; missing local file
+ errors; an http MIRROR still routes to curl.
+- **`.txz`-set hash**: stable across file reordering; changes when a file's
+ content changes.
+- **variant guard**: a missing mount dir yields non-zero.
+
+The real end-to-end chain is verified by a manual first run on the VM (the user
+runs it and reports back), the same division of labor as `.extras/test-build`
+Task 10.
+
+## Layout
+
+```
+.extras/image-builder/
+├── config REGISTRY, MIRROR, VARIANTS, PKGDIR
+├── lib.sh shared: _log/_warn/_err, fetch, txz_hash, guards
+├── bootstrap.sh base image (root; installpkg from NAS)
+├── build-full-image.sh full image (slackpkg all series)
+├── build-sbo-testbuild.sh testbuild image (installpkg sbopkg + tools)
+├── test-image-builder.sh pure-logic self-check
+└── README VM-side setup, reproducible
+```
+
+Lives in `.extras/` for now (per the user), relocatable later.
+
+The README carries the user-side checklist so it is reproducible and not
+trapped in chat:
+
+- VM specs (4/4/80), Slackware x86_64, docker installed.
+- Two NFS mounts (ro), root-readable, named `slackware64-current` /
+ `slackware64-15.0` under `/mnt/nas`.
+- `registry:2` container: `-v /opt/registry/data:/var/lib/registry`,
+ `--restart=always`, port 5000.
+- `/etc/docker/daemon.json` insecure-registry entry on the VM **and** on every
+ pulling client (this dev box, `buildsystem`), docker restarted.
+- Drop `sbopkg-*.txz` and `sbo-maintainer-tools-*.txz` into
+ `/opt/sbo-testbuild/pkgs/`.
+- Install the three cron lines.
+- VM static IP / LAN DNS for `docker.noland.dnx`.
+
+## Out of scope
+
+- i586 images (extensible later via `VARIANTS` / arch machinery).
+- Building sbopkg / sbo-maintainer-tools from source in the image (installed
+ from prebuilt `.txz`).
+- The forge `build-builder-image.sh` toolchain layer (redundant with full).
+- `installed_in_base` implementation in `.extras/test-build` — unblocked by
+ this work (queries the built image's package db) but tracked separately.
+- Registry auth / TLS (LAN-only plain HTTP).