image-builder: daily registry GC, registry on its own diskHEADmaster

On sda2 the registry grew 13G -> 53G between weekly GCs and filled /.
That broke /tmp, the build log, buildx state and the GC itself, so every
build failed from 2026-10-04. Run registry-gc.sh daily at 07:30 instead
of Sundays only.

The store has since moved to a dedicated disk with backup=0, out of
vzdump. Record that in fstab.example along with two traps hit during
the fix: `crontab -` on a full disk silently writes a 0-byte crontab,
and a plain docker stop/start leaves the registry serving the old,
deleted directory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

3 files changed, 21 insertions, 12 deletions

diff --git a/image-builder/README b/image-builder/README
index a1ea0c4..2594e30 100644
--- a/image-builder/README
+++ b/image-builder/README
@@ -104,12 +104,12 @@ cleanups keep it bounded:
 
   docker image prune -f   (daily) removes dangling images left in the docker
                           store when a tag moves to a freshly built image.
-  registry-gc.sh          (weekly) reclaims unreferenced blobs from the
+  registry-gc.sh          (daily) reclaims unreferenced blobs from the
                           registry's own store.
 
 registry-gc.sh is deliberately conservative:
   * it refuses to run while any build script is active, so it can never race a
-    push (cron runs it at 08:00 Sunday, well after the ~06:30 chain);
+    push (cron runs it daily at 07:30, after the ~06:30 chain);
   * it stops the registry so the manifest/blob graph is stable, and restarts it
     via an EXIT trap even if collection fails part-way;
   * it deletes only untagged manifests (-m) and the blobs they alone
diff --git a/image-builder/crontab.example b/image-builder/crontab.example
index 28f72d1..2b2cc24 100644
--- a/image-builder/crontab.example
+++ b/image-builder/crontab.example
@@ -17,7 +17,7 @@
 #   03:00  -current   bootstrap -> full -> testbuild
 #   05:00  15.0       bootstrap -> full -> testbuild
 #   07:00  reclaim    dangling images (catches both variants)
-#   08:00  registry   blob GC, Sundays only
+#   07:30  registry   blob GC, daily
 #   09:00  staleness  alert if any tag stopped moving
 #
 # Repos sync at 01:00/02:00, so the chain starts after that and the images are
@@ -68,9 +68,11 @@
 # build). Cheap when there is nothing to do.
 0  7 * * *   docker image prune -f >> /var/log/sbo-testbuild.log 2>&1
 # The registry never reclaims on its own: every push adds blobs and nothing
-# removes them, so its store grows until the disk fills. Weekly is enough.
+# removes them, so its store grows until the disk fills. Weekly was enough
+# while the store lived on sdb1; on sda2 (since 2026-09-22) it has ~55G and
+# grew 13G -> 53G in one week, filling / and failing every build. Daily.
 # registry-gc.sh has its own safety gates; see the script.
-0  8 * * 0   /opt/sbo-testbuild/image-builder/notify.sh run "registry GC" /opt/sbo-testbuild/image-builder/registry-gc.sh >> /var/log/sbo-testbuild.log 2>&1
+30 7 * * *   /opt/sbo-testbuild/image-builder/notify.sh run "registry GC" /opt/sbo-testbuild/image-builder/registry-gc.sh >> /var/log/sbo-testbuild.log 2>&1
 
 # ---------------------------------------------------------------------------
 # Staleness check
diff --git a/image-builder/fstab.example b/image-builder/fstab.example
index b0ca1d9..5e63c35 100644
--- a/image-builder/fstab.example
+++ b/image-builder/fstab.example
@@ -21,7 +21,7 @@
 UUID=0e5d008d-0ee0-41bc-9222-aedba1e088d0  /var/lib/docker  ext4  defaults  0  2
 
 # ---------------------------------------------------------------------------
-# Registry store: NOT on the docker disk
+# Registry store: its own disk, NOT the docker or system disk
 # ---------------------------------------------------------------------------
 # The registry's blob store used to be a bind mount from the docker disk
 # (/var/lib/docker/registry-data). That put ~25G of permanently-resident data
@@ -29,19 +29,26 @@ UUID=0e5d008d-0ee0-41bc-9222-aedba1e088d0  /var/lib/docker  ext4  defaults  0  2
 # registry grows with every push, the build needs headroom at 03:20, and the
 # build lost. Moved to the system disk on 2026-09-22, which had 49G idle.
 #
-# Keep them separate. The registry is small, static and I/O-light; the build
-# disk is large, churning and latency-insensitive. Sharing one volume couples
-# a slow leak to a hard failure.
+# That traded one shared volume for another. Between weekly GCs the registry
+# grew 13G -> 53G and filled /, which took down /tmp, the log, buildx state
+# and even crontab (a `crontab -` on the full disk wrote a 0-byte file). On
+# 2026-10-05 it moved to its own disk with backup=0 in Proxmox, so it is out
+# of vzdump and a leak can only fill itself.
 #
-# If this host's backups cover the system disk, exclude /opt/registry-data:
-# the contents are reproducible by re-pushing the images.
+# Keep it separate from both. The registry is small, static and I/O-light;
+# the build disk is large, churning and latency-insensitive; the system disk
+# must never fill. Sharing a volume couples a slow leak to a hard failure.
+# The contents are reproducible by re-pushing the images, so never back it up.
+UUID=<registry-disk-uuid>  /opt/registry-data  ext4  defaults  0  2
 #
 # Moving it is not just an fstab edit. dockerd caches the mount in its own
 # namespace, so after remounting you must restart the daemon and recreate the
 # registry container, or pushes keep silently landing on the old disk. Verify
 # with:  grep ' /var/lib/registry ' /proc/$(docker inspect registry \
 #            --format '{{.State.Pid}}')/mountinfo
-# and check the device is the system disk, not the docker one.
+# and check the device is the registry disk. A plain `docker stop/start`
+# is not enough: on 2026-10-05 the restarted container still saw the old,
+# deleted directory on the system disk and served an empty store.
 /opt/registry-data  /opt/sbo-testbuild/registry  none  bind  0  0
 
 # ---------------------------------------------------------------------------