image-builder: add safe registry GC and stop OCI-index breakageHEADmaster

The registry never reclaims blobs, so its store grows until the disk fills and the nightly builds fail with "no space left on device". Add registry-gc.sh, run weekly (Sunday 08:00), plus a daily dangling-image prune.

registry-gc.sh refuses to run while a build is active, stops the registry for a stable blob graph, deletes only untagged manifests (-m) and their blobs, restarts via an EXIT trap, and verifies a tag still pulls.

distribution 2.8.x GC does not follow OCI image indexes, so -m deletes their child manifests (distribution#3178). Default BuildKit provenance made every pushed tag an OCI index, which made -m destructive. Build scripts now pass --provenance=false (plain schema2), and registry-gc.sh refuses to run if any tag is still an index.

1 files changed, 52 insertions, 3 deletions

diff --git a/image-builder/README b/image-builder/README
index 8711b1d..86c463f 100644
--- a/image-builder/README
+++ b/image-builder/README
@@ -10,6 +10,9 @@ Three scripts, chained (see docs/specs/2026-07-13-image-builder-design.md):
   build-full-image.sh    sbo-full:{ver}       FROM base, all series
   build-sbo-testbuild.sh sbo-testbuild:{ver}  FROM full, + sbopkg + tools
 
+Plus one maintenance script (not part of the chain):
+  registry-gc.sh         reclaim unreferenced blobs from the registry store
+
 All settings live in ./config.
 
 VM setup (docker.noland.dnx, Slackware x86_64, 4 vCPU / 4 GB / 80 GB)
@@ -22,9 +25,9 @@ VM setup (docker.noland.dnx, Slackware x86_64, 4 vCPU / 4 GB / 80 GB)
    Each is a full mirror (PACKAGES.TXT, ChangeLog.txt, slackware64/, patches/,
    extra/). Root must be able to read them (bootstrap runs installpkg as root).
 
-3. Run a LAN registry:
+3. Run a LAN registry (storage on the same disk as docker, bind-mounted):
      docker run -d --restart=always -p 5000:5000 \
-       -v /opt/registry/data:/var/lib/registry --name registry registry:2
+       -v /opt/sbo-testbuild/registry:/var/lib/registry --name registry registry:2
 
 4. Mark the registry insecure (plain HTTP) on the VM AND every pulling client
    (this dev box, the buildsystem VM). In /etc/docker/daemon.json:
@@ -49,7 +52,18 @@ VM setup (docker.noland.dnx, Slackware x86_64, 4 vCPU / 4 GB / 80 GB)
       # 15.0 (ready ~06:35)
       0  5 * * *  /path/to/sbo-dockerbuild/image-builder/bootstrap.sh           --version 15.0 >> /var/log/sbo-testbuild.log 2>&1
      20  5 * * *  /path/to/sbo-dockerbuild/image-builder/build-full-image.sh    --version 15.0 >> /var/log/sbo-testbuild.log 2>&1
-     30  6 * * *  /path/to/sbo-dockerbuild/image-builder/build-sbo-testbuild.sh --version 15.0 >> /var/log/sbo-testbuild.log 2>&1
+      30 6 * * *  /path/to/sbo-dockerbuild/image-builder/build-sbo-testbuild.sh --version 15.0 >> /var/log/sbo-testbuild.log 2>&1
+
+   Post-build cleanup, after the chain (which ends ~06:30) and before the 15:00
+   cache prune:
+      # daily: drop dangling images left behind when a tag moves to a new build
+      0  7 * * *  docker image prune -f >> /var/log/sbo-testbuild.log 2>&1
+      # weekly (Sunday): reclaim unreferenced blobs from the registry store
+      0  8 * * 0  /path/to/sbo-dockerbuild/image-builder/registry-gc.sh >> /var/log/sbo-testbuild.log 2>&1
+
+   The registry never reclaims blobs on its own, so without the weekly GC its
+   storage grows until the disk fills and the nightly builds fail with
+   "no space left on device" (see the section below).
 
 7. Ensure docker.noland.dnx resolves on the LAN (static IP or DNS).
 
@@ -64,6 +78,41 @@ Then confirm:
 
 Flags: --force (rebuild unconditionally), --version <current|15.0> (one variant).
 
+Registry garbage collection (registry-gc.sh)
+--------------------------------------------
+The registry keeps every blob ever pushed; it never reclaims on its own. Left
+alone, the store grows until the disk fills and the nightly builds fail. Two
+cleanups keep it bounded:
+
+  docker image prune -f   (daily) removes dangling images left in the docker
+                          store when a tag moves to a freshly built image.
+  registry-gc.sh          (weekly) reclaims unreferenced blobs from the
+                          registry's own store.
+
+registry-gc.sh is deliberately conservative:
+  * it refuses to run while any build script is active, so it can never race a
+    push (cron runs it at 08:00 Sunday, well after the ~06:30 chain);
+  * it stops the registry so the manifest/blob graph is stable, and restarts it
+    via an EXIT trap even if collection fails part-way;
+  * it deletes only untagged manifests (-m) and the blobs they alone
+    reference, so every tag keeps resolving;
+  * it verifies afterwards that a tag still pulls.
+
+Why the build scripts pass --provenance=false: with default BuildKit
+provenance, `docker push` stores an OCI image index (the image plus an
+attestation manifest). Distribution 2.8.x garbage collection does not follow
+OCI indexes, so `-m` would delete their child manifests and orphan the layer
+blobs (distribution issue #3178). Disabling provenance keeps each tag a plain
+Docker schema2 manifest, which the collector handles correctly. registry-gc.sh
+refuses to run if it finds any tag that is still an index, so this cannot
+regress silently.
+
+Preview without touching anything (registry stays up, nothing is deleted):
+  ./registry-gc.sh --dry-run
+
+Storage path is resolved from the running container's /var/lib/registry mount,
+so the script follows the registry wherever it is mounted.
+
 Tests
 -----
    bash test-image-builder.sh     # pure-logic self-check, no docker