diff options
Diffstat (limited to 'image-builder/crontab.example')
| -rw-r--r-- | image-builder/crontab.example | 33 |
1 files changed, 26 insertions, 7 deletions
diff --git a/image-builder/crontab.example b/image-builder/crontab.example index 174ace0..28f72d1 100644 --- a/image-builder/crontab.example +++ b/image-builder/crontab.example @@ -18,9 +18,14 @@ # 05:00 15.0 bootstrap -> full -> testbuild # 07:00 reclaim dangling images (catches both variants) # 08:00 registry blob GC, Sundays only +# 09:00 staleness alert if any tag stopped moving # # Repos sync at 01:00/02:00, so the chain starts after that and the images are # ready by 09:00. +# +# Every build runs under `notify.sh run`, which posts to Gotify on a non-zero +# exit and passes the status through. That is necessary but not sufficient: +# see the staleness check at the bottom for why. # --------------------------------------------------------------------------- # Pre-build reclaim @@ -47,13 +52,13 @@ # no-op that exits in seconds. # # -current (moves daily): -0 3 * * * /opt/sbo-testbuild/image-builder/bootstrap.sh --version current >> /var/log/sbo-testbuild.log 2>&1 -20 3 * * * /opt/sbo-testbuild/image-builder/build-full-image.sh --version current >> /var/log/sbo-testbuild.log 2>&1 -30 4 * * * /opt/sbo-testbuild/image-builder/build-sbo-testbuild.sh --version current >> /var/log/sbo-testbuild.log 2>&1 +0 3 * * * /opt/sbo-testbuild/image-builder/notify.sh run "bootstrap current" /opt/sbo-testbuild/image-builder/bootstrap.sh --version current >> /var/log/sbo-testbuild.log 2>&1 +20 3 * * * /opt/sbo-testbuild/image-builder/notify.sh run "full current" /opt/sbo-testbuild/image-builder/build-full-image.sh --version current >> /var/log/sbo-testbuild.log 2>&1 +30 4 * * * /opt/sbo-testbuild/image-builder/notify.sh run "testbuild current" /opt/sbo-testbuild/image-builder/build-sbo-testbuild.sh --version current >> /var/log/sbo-testbuild.log 2>&1 # 15.0 (frozen stable; rebuilds only on a real repo update): -0 5 * * * /opt/sbo-testbuild/image-builder/bootstrap.sh --version 15.0 >> /var/log/sbo-testbuild.log 2>&1 -20 5 * * * /opt/sbo-testbuild/image-builder/build-full-image.sh --version 15.0 >> /var/log/sbo-testbuild.log 2>&1 -30 6 * * * /opt/sbo-testbuild/image-builder/build-sbo-testbuild.sh --version 15.0 >> /var/log/sbo-testbuild.log 2>&1 +0 5 * * * /opt/sbo-testbuild/image-builder/notify.sh run "bootstrap 15.0" /opt/sbo-testbuild/image-builder/bootstrap.sh --version 15.0 >> /var/log/sbo-testbuild.log 2>&1 +20 5 * * * /opt/sbo-testbuild/image-builder/notify.sh run "full 15.0" /opt/sbo-testbuild/image-builder/build-full-image.sh --version 15.0 >> /var/log/sbo-testbuild.log 2>&1 +30 6 * * * /opt/sbo-testbuild/image-builder/notify.sh run "testbuild 15.0" /opt/sbo-testbuild/image-builder/build-sbo-testbuild.sh --version 15.0 >> /var/log/sbo-testbuild.log 2>&1 # --------------------------------------------------------------------------- # Post-build cleanup @@ -65,4 +70,18 @@ # The registry never reclaims on its own: every push adds blobs and nothing # removes them, so its store grows until the disk fills. Weekly is enough. # registry-gc.sh has its own safety gates; see the script. -0 8 * * 0 /opt/sbo-testbuild/image-builder/registry-gc.sh >> /var/log/sbo-testbuild.log 2>&1 +0 8 * * 0 /opt/sbo-testbuild/image-builder/notify.sh run "registry GC" /opt/sbo-testbuild/image-builder/registry-gc.sh >> /var/log/sbo-testbuild.log 2>&1 + +# --------------------------------------------------------------------------- +# Staleness check +# --------------------------------------------------------------------------- +# The exit-status alerts above would not have caught the September 2026 +# outage on their own. build-full-image.sh failed for ten nights and did +# report non-zero, but build-sbo-testbuild.sh exited 0 every single night: +# it saw an unchanged parent digest and skipped, which is correct. So after +# the first alert the chain went quiet while its tags aged six days. +# +# This asks the registry a different question: not "did anything error" but +# "is anything still current". It catches a skipped stage, a stopped cron and +# a wedged mirror alike. Runs after the chain has had its chance. +0 9 * * * /opt/sbo-testbuild/image-builder/notify.sh stale >> /var/log/sbo-testbuild.log 2>&1 |