diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/superpowers/specs/2026-06-24-package-cache-design.md | 222 |
1 files changed, 222 insertions, 0 deletions
diff --git a/docs/superpowers/specs/2026-06-24-package-cache-design.md b/docs/superpowers/specs/2026-06-24-package-cache-design.md new file mode 100644 index 0000000..70a073f --- /dev/null +++ b/docs/superpowers/specs/2026-06-24-package-cache-design.md @@ -0,0 +1,222 @@ +# Persistent package cache design + +Date: 2026-06-24 +Project: sbo-batch-test + +## Problem + +In a build run, the named target's dependency chain is built one package at a +time in a fresh overlay. Dependencies are rebuilt from scratch every run even +when they have not changed since a previous run. Building a leaf dep can take +minutes; rebuilding unchanged deps is wasted time. + +The fix is a persistent, on-disk cache of built packages. A dependency whose +version has not changed is installed from the cache instead of being rebuilt. +The package under test is always built fresh (it is the thing being verified). + +## Scope + +- Single-package runs: `sbo-batch-test <prog>`. The named prog is the target; + every other package in its resolved chain is a dependency. +- Category / "all" / queue modes remain stubbed TODOs and are out of scope. The + same `build_one` cache logic will apply when they are built later, with no + redesign needed (each invocation's CLI target builds fresh). + +## Decisions (settled) + +- Cache is persistent across runs, on local disk. +- Cache key is `prog + version`. Build number, arch, and tag are NOT part of the + match (single-arch repo, fixed tag; build number auto-increments per build and + is therefore noise for matching). +- The target (the CLI argument) ALWAYS builds fresh, even on a cache hit, and + its fresh output refreshes the cache. +- A dependency uses the cache on a version match (hit), otherwise it builds + fresh and is then cached (miss). +- Across runs, a cached package is reused as a dependency as long as its version + is unchanged. +- On store, the prog's cache directory is cleared first, so it holds exactly one + `.txz`: the latest tested build of the latest tested version. +- When `update_base` actually patches the base, the entire cache is wiped (cached + deps were built against the old base). + +## Configuration + +New config variable, set in the external config file (see `config.example`): + +```sh +# Persistent package cache. Local disk, survives across runs. Built dependency +# packages are stored here and reused when their version is unchanged. Wiped +# automatically when the base is patched. +PKG_CACHE="/var/cache/sbo-batch-test" +``` + +Defaults to empty in the script; if unset/empty the cache is disabled (every +package builds fresh, current behavior). `validate_env` does not hard-require it. + +## Cache layout + +Mirrors the SBo tree: + +``` +$PKG_CACHE/<category>/<prog>/<prog>-<version>-<arch>-<build>_<tag>.txz +``` + +Example: + +``` +/var/cache/sbo-batch-test/personal/claude-code-bin/claude-code-bin-2.1.140-x86_64-1_danix.txz +``` + +Each `<cat>/<prog>/` directory holds exactly one `.txz` (the latest tested +build). Eviction is therefore "clear the prog directory, then copy the new +`.txz` in", no sibling matching needed. + +## Core logic (pure, unit-testable) + +Three small functions, free of chroot/installpkg side effects so `test-logic.sh` +can cover them: + +### `cache_decision <cat> <prog> <version>` + +Globs `$PKG_CACHE/<cat>/<prog>/<prog>-*.t?z` (any version) and echoes one token: + +- `cached` — a `.txz` for exactly `<version>` is present. +- `bump:<oldver>:<newver>` — a `.txz` is present but for a different version + (`<oldver>` parsed from the cached filename, `<newver>` = requested version). +- `new` — nothing cached for this prog. + +Version is parsed from the filename by stripping the `<prog>-` prefix and taking +the field up to the next `-`. Glob is nullglob-safe (`[[ -e ]]` guard). Missing +`PKG_CACHE` or prog dir is treated as `new`. If `PKG_CACHE` is empty (disabled), +always returns `new`. + +### `cache_store <cat> <prog> <src_txz>` + +`mkdir -p` the prog dir, remove its existing contents, copy `<src_txz>` in. No-op +if `PKG_CACHE` is empty. + +### `cache_path <cat> <prog> <version>` + +Echoes the path of the cached `.txz` for `<cat>/<prog>` at `<version>` (the file +that made `cache_decision` return `cached`), for the installpkg-from-cache path. +Empty output if no hit. + +## Build-flow integration + +`build_one` gains an `is_target` argument (true only for the package whose dir +equals the run's target dir). `run_target` passes it: for each package in the +resolved chain, `is_target` is true iff its dir is the CLI target's dir. + +Decision at the top of `build_one`, given `cat`, `prog`, `version` (from +`.info`): + +- **Target (`is_target` true):** always take the build path. After a successful + build, `cache_store` the produced `.txz`. +- **Dependency (`is_target` false):** + - `cache_decision` == `cached` → cache-install path: copy the cached `.txz` + from the host into the overlay (`$c/sbo-work/`), `installpkg --terse` it in + the chroot, log it, set status `CACHED`. No download/build. + - `cache_decision` == `bump:OLD:NEW` or `new` → build path. After success, + `cache_store` the produced `.txz`. + +Caching across the overlay boundary: + +- The build runs inside the chroot; the produced `.txz` lands in the overlay's + `$OUTPUT`. To cache it, the host side copies it OUT of the overlay (from the + known overlay path) before teardown, into `PKG_CACHE` via `cache_store`. +- A cache hit copies the host-side cached `.txz` INTO the overlay, then + `installpkg --terse` runs in the chroot (same as a freshly built package). + +## Reporting + +### Per-package outcome (build order + per-package line) + +Annotate each package in the resolved order so the reason for build vs reuse is +visible. Shown in BOTH a real run and `--dry-run`: + +``` +build order: + libfoo cached (1.1) + libbar rebuild: 1.0 -> 1.1 + libbaz build (new) + progX target, rebuild: 1.0 -> 1.1 +``` + +- `cached (<ver>)` — dep hit, will be / was installed from cache. +- `rebuild: <old> -> <new>` — version bump; old cache evicted, built fresh. +- `build (new)` — nothing cached, built fresh. +- The target is prefixed `target,` and always shows a build outcome (it never + uses the cache); if a different version was cached it still reports the bump so + the version change is visible. + +This derives from `cache_decision` for every package, target included. + +### Status value + +New status `CACHED`, distinct from `SUCCESS`, set for a dependency installed from +the cache. The target is never `CACHED`. Added to the status list. + +### Summary + +Add a cached count to the summary line: + +``` +3 succeeded, 0 failed, 0 blocked, 2 cached, total 41s +``` + +## Dry-run + +`--dry-run` resolves and prints the annotated build order (above) using +`cache_decision` only. It performs NO installpkg, NO build, and NO cache write. +Read-only. Lets the user preview reuse vs rebuild before a real run. + +## Base-patch invalidation + +In `update_base`, inside the existing "patching" branch (taken only when the +mirror ChangeLog head differs from the recorded marker), after patching +succeeds, wipe the cache: + +```sh +rm -rf "${PKG_CACHE:?}"/* # base changed; cached deps are suspect +``` + +Guarded so it never runs with an empty `PKG_CACHE`. Normal runs (base +up-to-date) do not touch the cache. + +## Edge cases + +- `PKG_CACHE` empty/unset: cache disabled, every package builds fresh (current + behavior). No errors. +- Cache dir missing: treated as all-miss, created on first `cache_store`. +- Multiple `.txz` ever present in a prog dir (should not happen given eviction): + `cache_decision` and `cache_path` take the newest by mtime. +- Package extensions: stored and matched as `*.t?z` (covers txz/tgz/tbz/tlz), + consistent with the existing build flow. +- A dynamic SlackBuild that produces an unexpected version: the produced `.txz` + is cached under whatever version it actually built; the next run's + `cache_decision` compares against `.info`, so a mismatch simply rebuilds. Safe, + never a false hit. + +## Testing + +Extend `test-logic.sh` (no VM needed) to cover the pure functions against a +seeded fake `$PKG_CACHE` tree: + +- `cache_decision` returns `cached` on exact version match. +- `cache_decision` returns `bump:OLD:NEW` when a different version is cached. +- `cache_decision` returns `new` for an empty/absent prog dir. +- `cache_decision` returns `new` when `PKG_CACHE` is empty (disabled). +- `cache_store` results in exactly one `.txz` in the prog dir (eviction). +- `cache_path` returns the hit file for a matching version. + +VM-only (out of self-check reach, by design, same boundary as the rest of the +build flow): installpkg of a cached `.txz` in the chroot, copy-out-of-overlay on +store, and the base-patch wipe firing. These mirror the reference build flow. + +## Out of scope / not changed + +- No network resolution (hard constraint). +- No slackrepo interaction; the slackrepo hintfiles under `/etc/slackrepo` are a + separate workflow and are not read by this tool. +- Category/"all"/queue modes and `-j` parallelism remain stubbed. +- `--keep` is not reintroduced. |
