diff options
Diffstat (limited to 'image-builder/README')
| -rw-r--r-- | image-builder/README | 55 |
1 files changed, 52 insertions, 3 deletions
diff --git a/image-builder/README b/image-builder/README index 8711b1d..86c463f 100644 --- a/image-builder/README +++ b/image-builder/README @@ -10,6 +10,9 @@ Three scripts, chained (see docs/specs/2026-07-13-image-builder-design.md): build-full-image.sh sbo-full:{ver} FROM base, all series build-sbo-testbuild.sh sbo-testbuild:{ver} FROM full, + sbopkg + tools +Plus one maintenance script (not part of the chain): + registry-gc.sh reclaim unreferenced blobs from the registry store + All settings live in ./config. VM setup (docker.noland.dnx, Slackware x86_64, 4 vCPU / 4 GB / 80 GB) @@ -22,9 +25,9 @@ VM setup (docker.noland.dnx, Slackware x86_64, 4 vCPU / 4 GB / 80 GB) Each is a full mirror (PACKAGES.TXT, ChangeLog.txt, slackware64/, patches/, extra/). Root must be able to read them (bootstrap runs installpkg as root). -3. Run a LAN registry: +3. Run a LAN registry (storage on the same disk as docker, bind-mounted): docker run -d --restart=always -p 5000:5000 \ - -v /opt/registry/data:/var/lib/registry --name registry registry:2 + -v /opt/sbo-testbuild/registry:/var/lib/registry --name registry registry:2 4. Mark the registry insecure (plain HTTP) on the VM AND every pulling client (this dev box, the buildsystem VM). In /etc/docker/daemon.json: @@ -49,7 +52,18 @@ VM setup (docker.noland.dnx, Slackware x86_64, 4 vCPU / 4 GB / 80 GB) # 15.0 (ready ~06:35) 0 5 * * * /path/to/sbo-dockerbuild/image-builder/bootstrap.sh --version 15.0 >> /var/log/sbo-testbuild.log 2>&1 20 5 * * * /path/to/sbo-dockerbuild/image-builder/build-full-image.sh --version 15.0 >> /var/log/sbo-testbuild.log 2>&1 - 30 6 * * * /path/to/sbo-dockerbuild/image-builder/build-sbo-testbuild.sh --version 15.0 >> /var/log/sbo-testbuild.log 2>&1 + 30 6 * * * /path/to/sbo-dockerbuild/image-builder/build-sbo-testbuild.sh --version 15.0 >> /var/log/sbo-testbuild.log 2>&1 + + Post-build cleanup, after the chain (which ends ~06:30) and before the 15:00 + cache prune: + # daily: drop dangling images left behind when a tag moves to a new build + 0 7 * * * docker image prune -f >> /var/log/sbo-testbuild.log 2>&1 + # weekly (Sunday): reclaim unreferenced blobs from the registry store + 0 8 * * 0 /path/to/sbo-dockerbuild/image-builder/registry-gc.sh >> /var/log/sbo-testbuild.log 2>&1 + + The registry never reclaims blobs on its own, so without the weekly GC its + storage grows until the disk fills and the nightly builds fail with + "no space left on device" (see the section below). 7. Ensure docker.noland.dnx resolves on the LAN (static IP or DNS). @@ -64,6 +78,41 @@ Then confirm: Flags: --force (rebuild unconditionally), --version <current|15.0> (one variant). +Registry garbage collection (registry-gc.sh) +-------------------------------------------- +The registry keeps every blob ever pushed; it never reclaims on its own. Left +alone, the store grows until the disk fills and the nightly builds fail. Two +cleanups keep it bounded: + + docker image prune -f (daily) removes dangling images left in the docker + store when a tag moves to a freshly built image. + registry-gc.sh (weekly) reclaims unreferenced blobs from the + registry's own store. + +registry-gc.sh is deliberately conservative: + * it refuses to run while any build script is active, so it can never race a + push (cron runs it at 08:00 Sunday, well after the ~06:30 chain); + * it stops the registry so the manifest/blob graph is stable, and restarts it + via an EXIT trap even if collection fails part-way; + * it deletes only untagged manifests (-m) and the blobs they alone + reference, so every tag keeps resolving; + * it verifies afterwards that a tag still pulls. + +Why the build scripts pass --provenance=false: with default BuildKit +provenance, `docker push` stores an OCI image index (the image plus an +attestation manifest). Distribution 2.8.x garbage collection does not follow +OCI indexes, so `-m` would delete their child manifests and orphan the layer +blobs (distribution issue #3178). Disabling provenance keeps each tag a plain +Docker schema2 manifest, which the collector handles correctly. registry-gc.sh +refuses to run if it finds any tag that is still an index, so this cannot +regress silently. + +Preview without touching anything (registry stays up, nothing is deleted): + ./registry-gc.sh --dry-run + +Storage path is resolved from the running container's /var/lib/registry mount, +so the script follows the registry wherever it is mounted. + Tests ----- bash test-image-builder.sh # pure-logic self-check, no docker |