|
The chain had no failure notification at all, which is why the September
outage ran ten days: every failure was in the log from the first night
and nobody read the log.
notify.sh has two modes because the chain fails in two ways and only one
of them has a non-zero exit status:
run <label> <cmd...> runs the command, posts on non-zero, and passes
the status through so cron still sees the truth.
stale posts if any tag is older than its budget.
The second exists because exit status alone would not have caught what
happened. build-full-image.sh did report non-zero for ten nights, but
build-sbo-testbuild.sh exited 0 every one of them: it saw an unchanged
parent digest and skipped, which is correct behaviour. After the first
alert the chain would have gone quiet while its tags aged six days. The
staleness check asks the registry a different question, "is anything
still current", and catches a skipped stage, a stopped cron or a wedged
mirror alike.
Budgets are split: -current rebuilds nightly (2 days), 15.0 is frozen
and legitimately sits still for weeks (30). A single budget would either
cry wolf on the stable tags or go blind on the rolling ones.
The token is read from /root/.gotify-token (mode 600, not in the repo).
Posting is best-effort throughout: a notifier that fails a build because
the notifier is down would be worse than no notifier.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
