From 889daacdf8e3a9f1f3a41fe3b546deff1bfbd0ae Mon Sep 17 00:00:00 2001 From: "Danilo M." Date: Mon, 5 Oct 2026 09:59:51 +0200 Subject: image-builder: daily registry GC, registry on its own disk On sda2 the registry grew 13G -> 53G between weekly GCs and filled /. That broke /tmp, the build log, buildx state and the GC itself, so every build failed from 2026-10-04. Run registry-gc.sh daily at 07:30 instead of Sundays only. The store has since moved to a dedicated disk with backup=0, out of vzdump. Record that in fstab.example along with two traps hit during the fix: `crontab -` on a full disk silently writes a 0-byte crontab, and a plain docker stop/start leaves the registry serving the old, deleted directory. Co-Authored-By: Claude Opus 5.5 --- image-builder/fstab.example | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) (limited to 'image-builder/fstab.example') diff --git a/image-builder/fstab.example b/image-builder/fstab.example index b0ca1d9..5e63c35 100644 --- a/image-builder/fstab.example +++ b/image-builder/fstab.example @@ -21,7 +21,7 @@ UUID=0e5d008d-0ee0-41bc-9222-aedba1e088d0 /var/lib/docker ext4 defaults 0 2 # --------------------------------------------------------------------------- -# Registry store: NOT on the docker disk +# Registry store: its own disk, NOT the docker or system disk # --------------------------------------------------------------------------- # The registry's blob store used to be a bind mount from the docker disk # (/var/lib/docker/registry-data). That put ~25G of permanently-resident data @@ -29,19 +29,26 @@ UUID=0e5d008d-0ee0-41bc-9222-aedba1e088d0 /var/lib/docker ext4 defaults 0 2 # registry grows with every push, the build needs headroom at 03:20, and the # build lost. Moved to the system disk on 2026-09-22, which had 49G idle. # -# Keep them separate. The registry is small, static and I/O-light; the build -# disk is large, churning and latency-insensitive. Sharing one volume couples -# a slow leak to a hard failure. +# That traded one shared volume for another. Between weekly GCs the registry +# grew 13G -> 53G and filled /, which took down /tmp, the log, buildx state +# and even crontab (a `crontab -` on the full disk wrote a 0-byte file). On +# 2026-10-05 it moved to its own disk with backup=0 in Proxmox, so it is out +# of vzdump and a leak can only fill itself. # -# If this host's backups cover the system disk, exclude /opt/registry-data: -# the contents are reproducible by re-pushing the images. +# Keep it separate from both. The registry is small, static and I/O-light; +# the build disk is large, churning and latency-insensitive; the system disk +# must never fill. Sharing a volume couples a slow leak to a hard failure. +# The contents are reproducible by re-pushing the images, so never back it up. +UUID= /opt/registry-data ext4 defaults 0 2 # # Moving it is not just an fstab edit. dockerd caches the mount in its own # namespace, so after remounting you must restart the daemon and recreate the # registry container, or pushes keep silently landing on the old disk. Verify # with: grep ' /var/lib/registry ' /proc/$(docker inspect registry \ # --format '{{.State.Pid}}')/mountinfo -# and check the device is the system disk, not the docker one. +# and check the device is the registry disk. A plain `docker stop/start` +# is not enough: on 2026-10-05 the restarted container still saw the old, +# deleted directory on the system disk and served an empty store. /opt/registry-data /opt/sbo-testbuild/registry none bind 0 0 # --------------------------------------------------------------------------- -- cgit v1.2.3