sbo-testbuild image builder =========================== Builds the docker images that test-build consumes: docker.noland.dnx:5000/sbo-testbuild:current docker.noland.dnx:5000/sbo-testbuild:15.0 Three scripts, chained (see docs/specs/2026-07-13-image-builder-design.md): bootstrap.sh sbo-base:{ver} FROM scratch, base pkgs from NAS build-full-image.sh sbo-full:{ver} FROM base, all series build-sbo-testbuild.sh sbo-testbuild:{ver} FROM full, + sbopkg + tools Plus one maintenance script (not part of the chain): registry-gc.sh reclaim unreferenced blobs from the registry store All settings live in ./config. VM setup (docker.noland.dnx, Slackware x86_64, 4 vCPU / 4 GB / 80 GB) -------------------------------------------------------------------- 1. Install docker; enable the daemon. 2. NFS-mount the two NAS trees read-only, named to match: /mnt/nas/slackware64-current -> -current mirror tree /mnt/nas/slackware64-15.0 -> 15.0 mirror tree Each is a full mirror (PACKAGES.TXT, ChangeLog.txt, slackware64/, patches/, extra/). Root must be able to read them (bootstrap runs installpkg as root). 3. Run a LAN registry (storage on the same disk as docker, bind-mounted): docker run -d --restart=always -p 5000:5000 \ -v /opt/sbo-testbuild/registry:/var/lib/registry --name registry registry:2 4. Mark the registry insecure (plain HTTP) on the VM AND every pulling client (this dev box, the buildsystem VM). In /etc/docker/daemon.json: { "insecure-registries": ["docker.noland.dnx:5000"] } then restart docker. 5. Drop the two prebuilt packages (built once, re-drop on upstream bumps): /opt/sbo-testbuild/pkgs/sbopkg-*.txz /opt/sbo-testbuild/pkgs/sbo-maintainer-tools-*.txz 6. Install the nightly cron (root). The NAS repos sync at 01:00 and 02:00, so the chain runs after and both variants are ready well before the ~09:00 work start. No --force: each script self-gates (bootstrap on the ChangeLog hash, full-image on the base-image digest, build-sbo-testbuild on the full-image digest + tools .txz hash), so an unchanged night is a cheap no-op. -current moves daily and rebuilds most nights; 15.0 is frozen stable and rebuilds only on a real repo update. Deployed schedule on docker.noland.dnx: # -current (ready ~04:35) 0 3 * * * /path/to/sbo-dockerbuild/image-builder/bootstrap.sh --version current >> /var/log/sbo-testbuild.log 2>&1 20 3 * * * /path/to/sbo-dockerbuild/image-builder/build-full-image.sh --version current >> /var/log/sbo-testbuild.log 2>&1 30 4 * * * /path/to/sbo-dockerbuild/image-builder/build-sbo-testbuild.sh --version current >> /var/log/sbo-testbuild.log 2>&1 # 15.0 (ready ~06:35) 0 5 * * * /path/to/sbo-dockerbuild/image-builder/bootstrap.sh --version 15.0 >> /var/log/sbo-testbuild.log 2>&1 20 5 * * * /path/to/sbo-dockerbuild/image-builder/build-full-image.sh --version 15.0 >> /var/log/sbo-testbuild.log 2>&1 30 6 * * * /path/to/sbo-dockerbuild/image-builder/build-sbo-testbuild.sh --version 15.0 >> /var/log/sbo-testbuild.log 2>&1 Post-build cleanup, after the chain (which ends ~06:30) and before the 15:00 cache prune: # daily: drop dangling images left behind when a tag moves to a new build 0 7 * * * docker image prune -f >> /var/log/sbo-testbuild.log 2>&1 # weekly (Sunday): reclaim unreferenced blobs from the registry store 0 8 * * 0 /path/to/sbo-dockerbuild/image-builder/registry-gc.sh >> /var/log/sbo-testbuild.log 2>&1 The registry never reclaims blobs on its own, so without the weekly GC its storage grows until the disk fills and the nightly builds fail with "no space left on device" (see the section below). 7. Ensure docker.noland.dnx resolves on the LAN (static IP or DNS). Manual first run ---------------- ./bootstrap.sh --version current --force ./build-full-image.sh --version current --force ./build-sbo-testbuild.sh --version current --force Then confirm: docker pull docker.noland.dnx:5000/sbo-testbuild:current docker run --rm docker.noland.dnx:5000/sbo-testbuild:current sbopkg -V Flags: --force (rebuild unconditionally), --version (one variant). Registry garbage collection (registry-gc.sh) -------------------------------------------- The registry keeps every blob ever pushed; it never reclaims on its own. Left alone, the store grows until the disk fills and the nightly builds fail. Two cleanups keep it bounded: docker image prune -f (daily) removes dangling images left in the docker store when a tag moves to a freshly built image. registry-gc.sh (weekly) reclaims unreferenced blobs from the registry's own store. registry-gc.sh is deliberately conservative: * it refuses to run while any build script is active, so it can never race a push (cron runs it at 08:00 Sunday, well after the ~06:30 chain); * it stops the registry so the manifest/blob graph is stable, and restarts it via an EXIT trap even if collection fails part-way; * it deletes only untagged manifests (-m) and the blobs they alone reference, so every tag keeps resolving; * it verifies afterwards that a tag still pulls. Why the build scripts pass --provenance=false: with default BuildKit provenance, `docker push` stores an OCI image index (the image plus an attestation manifest). Distribution 2.8.x garbage collection does not follow OCI indexes, so `-m` would delete their child manifests and orphan the layer blobs (distribution issue #3178). Disabling provenance keeps each tag a plain Docker schema2 manifest, which the collector handles correctly. registry-gc.sh refuses to run if it finds any tag that is still an index, so this cannot regress silently. Preview without touching anything (registry stays up, nothing is deleted): ./registry-gc.sh --dry-run Storage path is resolved from the running container's /var/lib/registry mount, so the script follows the registry wherever it is mounted. Tests ----- bash test-image-builder.sh # pure-logic self-check, no docker