image-builder: alert to Gotify on failure and on staleness

The chain had no failure notification at all, which is why the September
outage ran ten days: every failure was in the log from the first night
and nobody read the log.

notify.sh has two modes because the chain fails in two ways and only one
of them has a non-zero exit status:

  run <label> <cmd...>  runs the command, posts on non-zero, and passes
                        the status through so cron still sees the truth.
  stale                 posts if any tag is older than its budget.

The second exists because exit status alone would not have caught what
happened. build-full-image.sh did report non-zero for ten nights, but
build-sbo-testbuild.sh exited 0 every one of them: it saw an unchanged
parent digest and skipped, which is correct behaviour. After the first
alert the chain would have gone quiet while its tags aged six days. The
staleness check asks the registry a different question, "is anything
still current", and catches a skipped stage, a stopped cron or a wedged
mirror alike.

Budgets are split: -current rebuilds nightly (2 days), 15.0 is frozen
and legitimately sits still for weeks (30). A single budget would either
cry wolf on the stable tags or go blind on the rolling ones.

The token is read from /root/.gotify-token (mode 600, not in the repo).
Posting is best-effort throughout: a notifier that fails a build because
the notifier is down would be worse than no notifier.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

1 files changed, 26 insertions, 7 deletions

diff --git a/image-builder/crontab.example b/image-builder/crontab.example
index 174ace0..28f72d1 100644
--- a/image-builder/crontab.example
+++ b/image-builder/crontab.example
@@ -18,9 +18,14 @@
 #   05:00  15.0       bootstrap -> full -> testbuild
 #   07:00  reclaim    dangling images (catches both variants)
 #   08:00  registry   blob GC, Sundays only
+#   09:00  staleness  alert if any tag stopped moving
 #
 # Repos sync at 01:00/02:00, so the chain starts after that and the images are
 # ready by 09:00.
+#
+# Every build runs under `notify.sh run`, which posts to Gotify on a non-zero
+# exit and passes the status through. That is necessary but not sufficient:
+# see the staleness check at the bottom for why.
 
 # ---------------------------------------------------------------------------
 # Pre-build reclaim
@@ -47,13 +52,13 @@
 # no-op that exits in seconds.
 #
 # -current (moves daily):
-0  3 * * * /opt/sbo-testbuild/image-builder/bootstrap.sh           --version current >> /var/log/sbo-testbuild.log 2>&1
-20 3 * * * /opt/sbo-testbuild/image-builder/build-full-image.sh    --version current >> /var/log/sbo-testbuild.log 2>&1
-30 4 * * * /opt/sbo-testbuild/image-builder/build-sbo-testbuild.sh --version current >> /var/log/sbo-testbuild.log 2>&1
+0  3 * * * /opt/sbo-testbuild/image-builder/notify.sh run "bootstrap current" /opt/sbo-testbuild/image-builder/bootstrap.sh           --version current >> /var/log/sbo-testbuild.log 2>&1
+20 3 * * * /opt/sbo-testbuild/image-builder/notify.sh run "full current"      /opt/sbo-testbuild/image-builder/build-full-image.sh    --version current >> /var/log/sbo-testbuild.log 2>&1
+30 4 * * * /opt/sbo-testbuild/image-builder/notify.sh run "testbuild current" /opt/sbo-testbuild/image-builder/build-sbo-testbuild.sh --version current >> /var/log/sbo-testbuild.log 2>&1
 # 15.0 (frozen stable; rebuilds only on a real repo update):
-0  5 * * * /opt/sbo-testbuild/image-builder/bootstrap.sh           --version 15.0 >> /var/log/sbo-testbuild.log 2>&1
-20 5 * * * /opt/sbo-testbuild/image-builder/build-full-image.sh    --version 15.0 >> /var/log/sbo-testbuild.log 2>&1
-30 6 * * * /opt/sbo-testbuild/image-builder/build-sbo-testbuild.sh --version 15.0 >> /var/log/sbo-testbuild.log 2>&1
+0  5 * * * /opt/sbo-testbuild/image-builder/notify.sh run "bootstrap 15.0"    /opt/sbo-testbuild/image-builder/bootstrap.sh           --version 15.0 >> /var/log/sbo-testbuild.log 2>&1
+20 5 * * * /opt/sbo-testbuild/image-builder/notify.sh run "full 15.0"         /opt/sbo-testbuild/image-builder/build-full-image.sh    --version 15.0 >> /var/log/sbo-testbuild.log 2>&1
+30 6 * * * /opt/sbo-testbuild/image-builder/notify.sh run "testbuild 15.0"    /opt/sbo-testbuild/image-builder/build-sbo-testbuild.sh --version 15.0 >> /var/log/sbo-testbuild.log 2>&1
 
 # ---------------------------------------------------------------------------
 # Post-build cleanup
@@ -65,4 +70,18 @@
 # The registry never reclaims on its own: every push adds blobs and nothing
 # removes them, so its store grows until the disk fills. Weekly is enough.
 # registry-gc.sh has its own safety gates; see the script.
-0  8 * * 0   /opt/sbo-testbuild/image-builder/registry-gc.sh >> /var/log/sbo-testbuild.log 2>&1
+0  8 * * 0   /opt/sbo-testbuild/image-builder/notify.sh run "registry GC" /opt/sbo-testbuild/image-builder/registry-gc.sh >> /var/log/sbo-testbuild.log 2>&1
+
+# ---------------------------------------------------------------------------
+# Staleness check
+# ---------------------------------------------------------------------------
+# The exit-status alerts above would not have caught the September 2026
+# outage on their own. build-full-image.sh failed for ten nights and did
+# report non-zero, but build-sbo-testbuild.sh exited 0 every single night:
+# it saw an unchanged parent digest and skipped, which is correct. So after
+# the first alert the chain went quiet while its tags aged six days.
+#
+# This asks the registry a different question: not "did anything error" but
+# "is anything still current". It catches a skipped stage, a stopped cron and
+# a wedged mirror alike. Runs after the chain has had its chance.
+0  9 * * *   /opt/sbo-testbuild/image-builder/notify.sh stale >> /var/log/sbo-testbuild.log 2>&1