sbo-testbuild image builder
===========================

Builds the docker images that test-build consumes:
  docker.noland.dnx:5000/sbo-testbuild:current
  docker.noland.dnx:5000/sbo-testbuild:15.0

Three scripts, chained (see docs/specs/2026-07-13-image-builder-design.md):
  bootstrap.sh           sbo-base:{ver}       FROM scratch, base pkgs from NAS
  build-full-image.sh    sbo-full:{ver}       FROM base, all series
  build-sbo-testbuild.sh sbo-testbuild:{ver}  FROM full, + sbopkg + tools

Plus one maintenance script (not part of the chain):
  registry-gc.sh         reclaim unreferenced blobs from the registry store

All settings live in ./config.

VM setup (docker.noland.dnx, Slackware x86_64, 4 vCPU / 4 GB / 80 GB)
--------------------------------------------------------------------
1. Install docker; enable the daemon.

2. NFS-mount the two NAS trees read-only, named to match:
     /mnt/nas/slackware64-current   -> -current mirror tree
     /mnt/nas/slackware64-15.0      -> 15.0 mirror tree
   Each is a full mirror (PACKAGES.TXT, ChangeLog.txt, slackware64/, patches/,
   extra/). Root must be able to read them (bootstrap runs installpkg as root).

3. Run a LAN registry (storage on the same disk as docker, bind-mounted):
     docker run -d --restart=always -p 5000:5000 \
       -v /opt/sbo-testbuild/registry:/var/lib/registry --name registry registry:2

4. Mark the registry insecure (plain HTTP) on the VM AND every pulling client
   (this dev box, the buildsystem VM). In /etc/docker/daemon.json:
     { "insecure-registries": ["docker.noland.dnx:5000"] }
   then restart docker.

5. Drop the two prebuilt packages (built once, re-drop on upstream bumps):
     /opt/sbo-testbuild/pkgs/sbopkg-*.txz
     /opt/sbo-testbuild/pkgs/sbo-maintainer-tools-*.txz

6. Install the nightly cron (root). The NAS repos sync at 01:00 and 02:00, so
   the chain runs after and both variants are ready well before the ~09:00 work
   start. No --force: each script self-gates (bootstrap on the ChangeLog hash,
   full-image on the base-image digest, build-sbo-testbuild on the full-image
   digest + tools .txz hash), so an unchanged night is a cheap no-op. -current
   moves daily and rebuilds most nights; 15.0 is frozen stable and rebuilds only
   on a real repo update. Deployed schedule on docker.noland.dnx:
      # -current (ready ~04:35)
      0  3 * * *  /path/to/sbo-dockerbuild/image-builder/bootstrap.sh           --version current >> /var/log/sbo-testbuild.log 2>&1
     20  3 * * *  /path/to/sbo-dockerbuild/image-builder/build-full-image.sh    --version current >> /var/log/sbo-testbuild.log 2>&1
     30  4 * * *  /path/to/sbo-dockerbuild/image-builder/build-sbo-testbuild.sh --version current >> /var/log/sbo-testbuild.log 2>&1
      # 15.0 (ready ~06:35)
      0  5 * * *  /path/to/sbo-dockerbuild/image-builder/bootstrap.sh           --version 15.0 >> /var/log/sbo-testbuild.log 2>&1
     20  5 * * *  /path/to/sbo-dockerbuild/image-builder/build-full-image.sh    --version 15.0 >> /var/log/sbo-testbuild.log 2>&1
      30 6 * * *  /path/to/sbo-dockerbuild/image-builder/build-sbo-testbuild.sh --version 15.0 >> /var/log/sbo-testbuild.log 2>&1

   Post-build cleanup, after the chain (which ends ~06:30) and before the 15:00
   cache prune:
      # daily: drop dangling images left behind when a tag moves to a new build
      0  7 * * *  docker image prune -f >> /var/log/sbo-testbuild.log 2>&1
      # weekly (Sunday): reclaim unreferenced blobs from the registry store
      0  8 * * 0  /path/to/sbo-dockerbuild/image-builder/registry-gc.sh >> /var/log/sbo-testbuild.log 2>&1

   The registry never reclaims blobs on its own, so without the weekly GC its
   storage grows until the disk fills and the nightly builds fail with
   "no space left on device" (see the section below).

7. Ensure docker.noland.dnx resolves on the LAN (static IP or DNS).

Manual first run
----------------
   ./bootstrap.sh --version current --force
   ./build-full-image.sh --version current --force
   ./build-sbo-testbuild.sh --version current --force
Then confirm:
   docker pull docker.noland.dnx:5000/sbo-testbuild:current
   docker run --rm docker.noland.dnx:5000/sbo-testbuild:current sbopkg -V

Flags: --force (rebuild unconditionally), --version <current|15.0> (one variant).

Registry garbage collection (registry-gc.sh)
--------------------------------------------
The registry keeps every blob ever pushed; it never reclaims on its own. Left
alone, the store grows until the disk fills and the nightly builds fail. Two
cleanups keep it bounded:

  docker image prune -f   (daily) removes dangling images left in the docker
                          store when a tag moves to a freshly built image.
  registry-gc.sh          (weekly) reclaims unreferenced blobs from the
                          registry's own store.

registry-gc.sh is deliberately conservative:
  * it refuses to run while any build script is active, so it can never race a
    push (cron runs it at 08:00 Sunday, well after the ~06:30 chain);
  * it stops the registry so the manifest/blob graph is stable, and restarts it
    via an EXIT trap even if collection fails part-way;
  * it deletes only untagged manifests (-m) and the blobs they alone
    reference, so every tag keeps resolving;
  * it verifies afterwards that a tag still pulls.

Why the build scripts pass --provenance=false: with default BuildKit
provenance, `docker push` stores an OCI image index (the image plus an
attestation manifest). Distribution 2.8.x garbage collection does not follow
OCI indexes, so `-m` would delete their child manifests and orphan the layer
blobs (distribution issue #3178). Disabling provenance keeps each tag a plain
Docker schema2 manifest, which the collector handles correctly. registry-gc.sh
refuses to run if it finds any tag that is still an index, so this cannot
regress silently.

Preview without touching anything (registry stays up, nothing is deleted):
  ./registry-gc.sh --dry-run

Storage path is resolved from the running container's /var/lib/registry mount,
so the script follows the registry wherever it is mounted.

Tests
-----
   bash test-image-builder.sh     # pure-logic self-check, no docker
