diff options
Diffstat (limited to 'docs/plans')
| -rw-r--r-- | docs/plans/2026-07-13-docker-test-build.md | 1579 | ||||
| -rw-r--r-- | docs/plans/2026-07-13-image-builder.md | 790 |
2 files changed, 2369 insertions, 0 deletions
diff --git a/docs/plans/2026-07-13-docker-test-build.md b/docs/plans/2026-07-13-docker-test-build.md new file mode 100644 index 0000000..6195de9 --- /dev/null +++ b/docs/plans/2026-07-13-docker-test-build.md @@ -0,0 +1,1579 @@ +# Docker test-build Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** A single bash script `.extras/test-build` that verifies an already-published SBo package still builds cleanly on `-current` or `15.0` inside a throwaway docker container, resolving and building its SBo deps from the local tree, caching built deps per image digest, and reporting per-package status + lint. + +**Architecture:** Host resolves the dep tree from the local SBo tree (logic ported from `sbo-batch-test`), applies current-vs-stable override rules, prints the build order and confirms, then runs one `docker run --rm` that builds deps (or installs them from a host cache) and the target, lints the target, and streams results back. The container is the disposable environment (no overlayfs). Images are built by a separate job and consumed by tag. + +**Tech Stack:** Bash, docker, Slackware `installpkg`/`makepkg`, `sbo-maintainer-tools` (`sbopkglint`). Self-check is plain bash (`test-logic.sh`), no framework. + +--- + +## Reference source + +Port from `/home/danix/Programming/GIT/sbo-batch-tester/sbo-batch-test`. These +functions are lifted with minimal change (they are pure or host-side): + +- `find_slackbuild_dir`, `category_of`, `pkg_key`, `read_requires`, + `version_of`, `installed_in_base` +- resolver: `_resolve_visit`, `resolve_target`, globals `RESOLVED_ORDER`, + `UNMET`, `CYCLES`, `HAS_README`, `_vstate` +- cache: `_cache_ver_of`, `cache_decision`, `cache_path`, `cache_store`, + `cache_label` +- `depends_on_failed`, `print_summary`, `lint_pkg`, `init_color` +- self-check patterns from + `/home/danix/Programming/GIT/sbo-batch-tester/test-logic.sh` + +What is NEW or CHANGED in this repo: + +- overlay lifecycle (`setup_overlay`/`teardown_overlay`/`cleanup_trap`) is + REMOVED. The container replaces it. +- `build_one` is rewritten to run its build steps via `docker run` instead of + `chroot`. +- image + tree selection by version (`--stable`). +- override engine (drop / rename / fetch) applied to the resolved order. +- cache namespaced by image digest. +- config keys differ (see Task 2). + +## File Structure + +- Create: `/home/danix/Programming/GIT/sbo-slackbuilds/.extras/test-build` + (the script, executable) +- Create: + `/home/danix/Programming/GIT/sbo-slackbuilds/.extras/test-build-config.example` + (config template) +- Create: + `/home/danix/Programming/GIT/sbo-slackbuilds/.extras/overrides.example` + (override-rules template) +- Create: + `/home/danix/Programming/GIT/sbo-slackbuilds/.extras/test-logic.sh` + (self-check) + +All work happens in `.extras/`. Run every command from the repo root +`/home/danix/Programming/GIT/sbo-slackbuilds`. + +Note on committing: this repo has a pre-commit hook that runs `sbolint` on +staged packages. `.extras/` files are not packages, but to be safe every +commit in this plan uses `SBOLINT=no git commit ...`. Commits are GPG-signed +automatically; do not disable signing. + +--- + +## Task 1: Scaffold the script + smoke self-check + +**Files:** +- Create: `.extras/test-build` +- Create: `.extras/test-logic.sh` + +- [ ] **Step 1: Write the failing self-check** + +Create `.extras/test-logic.sh`: + +```bash +#!/bin/bash +# +# Logic self-check for .extras/test-build. Covers the pure, host-side parts: +# dependency resolution, override application, unknown-dep -> UNMET, cache +# decisions, and BLOCKED-BY-DEP propagation. No docker. +# +# Run: bash .extras/test-logic.sh +# +set -uo pipefail + +SCRIPT="$(dirname "$0")/test-build" +T=$(mktemp -d) +BASE_DB=$(mktemp -d); mkdir -p "$BASE_DB" + +cleanup() { rm -rf "$T" "$BASE_DB"; } +trap cleanup EXIT + +# Source the script without running main(). Sourcing re-runs the CONFIG block, +# so set test vars AFTER the source. +LIB=$(mktemp) +sed '/^main "\$@"$/d' "$SCRIPT" > "$LIB" +# shellcheck disable=SC1090 +source "$LIB" 2>/dev/null +rm -f "$LIB" + +pass=0; fail=0 +ok() { echo " ok: $1"; ((pass++)); return 0; } +bad() { echo " FAIL: $1"; ((fail++)); return 0; } + +# Placeholder assertion so this file runs before any logic exists. +ok "script sources without executing main" + +echo +echo "$pass passed, $fail failed" +[[ $fail -eq 0 ]] || exit 1 +echo "ALL LOGIC CHECKS PASS" +``` + +- [ ] **Step 2: Run it to verify it fails** + +Run: `bash .extras/test-logic.sh` +Expected: FAIL. `$SCRIPT` does not exist yet, `sed`/`source` on a missing file +errors, no "ALL LOGIC CHECKS PASS". + +- [ ] **Step 3: Write the minimal script skeleton** + +Create `.extras/test-build`: + +```bash +#!/bin/bash +# +# test-build - verify an already-published SBo package still builds on a target +# Slackware version inside a throwaway docker container. Resolves + builds its +# SBo deps from the local tree, caches built deps per image digest, reports +# per-package status and lints the target. +# +# Dependency-resolution, cache, and summary logic are adapted from sbo-batch-test +# (github: danixland). The overlay chroot is replaced by a docker container: the +# container IS the disposable environment, so no overlayfs. +# +# No em dashes in prose by author convention. + +# ============================================================================= +# CONFIG (do not edit here; real values live in the external config file) +# ============================================================================= +SBO_TREE_CURRENT="" +SBO_TREE_STABLE="" +IMAGE_CURRENT="" +IMAGE_STABLE="" +LOG_ROOT="/var/log/sbo-test-build" +PKG_CACHE="" + +TB_CONFIG="${TB_CONFIG:-$HOME/.config/sbo-testbuild/config}" +if [[ -f "$TB_CONFIG" ]]; then + # shellcheck disable=SC1090 + source "$TB_CONFIG" +fi + +TB_OVERRIDES="${TB_OVERRIDES:-$HOME/.config/sbo-testbuild/overrides}" + +# ============================================================================= +set -uo pipefail +# Not -e: a package build failing is a handled outcome, not a script crash. + +main() { + : +} + +main "$@" +``` + +- [ ] **Step 4: Run the self-check to verify it passes** + +Run: `bash .extras/test-logic.sh` +Expected: PASS, ending `ALL LOGIC CHECKS PASS`. + +- [ ] **Step 5: Commit** + +```bash +chmod +x .extras/test-build +git add .extras/test-build .extras/test-logic.sh +SBOLINT=no git commit -m 'test-build: scaffold script + self-check' +``` + +--- + +## Task 2: Config, arg parsing, version selection + +**Files:** +- Modify: `.extras/test-build` +- Create: `.extras/test-build-config.example` +- Test: `.extras/test-logic.sh` + +- [ ] **Step 1: Write the failing test** + +Append to `.extras/test-logic.sh` before the `--- result ---` block (the +`echo; "$pass passed..."` lines at the end): + +```bash +# --- version selection ------------------------------------------------------ +SBO_TREE_CURRENT="/trees/current" +SBO_TREE_STABLE="/trees/stable" +IMAGE_CURRENT="sbo-testbuild:current" +IMAGE_STABLE="sbo-testbuild:15.0" + +VERSION_ID="current"; select_version_paths +[[ "$ACTIVE_TREE" == "/trees/current" ]] && ok "current -> current tree" || bad "current tree wrong: [$ACTIVE_TREE]" +[[ "$ACTIVE_IMAGE" == "sbo-testbuild:current" ]] && ok "current -> current image" || bad "current image wrong: [$ACTIVE_IMAGE]" + +VERSION_ID="15.0"; select_version_paths +[[ "$ACTIVE_TREE" == "/trees/stable" ]] && ok "15.0 -> stable tree" || bad "stable tree wrong: [$ACTIVE_TREE]" +[[ "$ACTIVE_IMAGE" == "sbo-testbuild:15.0" ]] && ok "15.0 -> stable image" || bad "stable image wrong: [$ACTIVE_IMAGE]" +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `bash .extras/test-logic.sh` +Expected: FAIL, `select_version_paths: command not found`. + +- [ ] **Step 3: Implement config globals, flags, parse_args, select_version_paths** + +In `.extras/test-build`, replace the `main() { : }` line and everything up to +`main "$@"` with the flags/globals and functions below (keep the CONFIG block +above untouched): + +```bash +# ---- flags / globals -------------------------------------------------------- +USE_COLOR=1 # --no-color or non-TTY disables +DRY_RUN=0 # --dry-run: resolve + print order, do not build +ASSUME_YES=0 # --yes: skip the confirm prompt (still prints the order) +USE_CACHE=1 # --no-cache disables the dep cache for one run +VERSION_ID="current" # "current" | "15.0"; set by --stable +TARGET_ARG="" + +ACTIVE_TREE="" # selected SBo tree (by version) +ACTIVE_IMAGE="" # selected image tag (by version) +RUN_DIR="" # timestamped log dir for this run + +# Status tracking. Keyed by "category/prog". Parallel assoc arrays. +declare -A ST_STATUS=() +declare -A ST_REASON=() +declare -A ST_TIME=() +declare -A ST_README=() + +usage() { + cat <<'EOF' +test-build - verify an SBo package builds on a target Slackware in docker + +USAGE: + test-build [OPTIONS] <program-name> + +OPTIONS: + -h, --help This text. + --stable Target Slackware 15.0 (image + tree). Default is -current. + --dry-run Resolve, apply overrides, print the build order, do not build. + --yes Skip the confirm prompt (the order is still printed first). + --no-cache Rebuild all deps this run, ignore/refresh the cache. + --no-color Disable ANSI color (auto-disabled when stdout is not a TTY). +EOF +} + +parse_args() { + while [[ $# -gt 0 ]]; do + case "$1" in + -h|--help) usage; exit 0 ;; + --stable|15.0) VERSION_ID="15.0"; shift ;; + --dry-run) DRY_RUN=1; shift ;; + --yes) ASSUME_YES=1; shift ;; + --no-cache) USE_CACHE=0; shift ;; + --no-color) USE_COLOR=0; shift ;; + -*) echo "Unknown option: $1" >&2; usage >&2; exit 2 ;; + *) + if [[ -n "$TARGET_ARG" ]]; then + echo "Only one target accepted (got '$TARGET_ARG' and '$1')." >&2 + exit 2 + fi + TARGET_ARG="$1"; shift ;; + esac + done + if [[ -z "$TARGET_ARG" ]]; then + echo "No target given." >&2; usage >&2; exit 2 + fi +} + +init_color() { + if [[ $USE_COLOR -eq 1 && -t 1 ]]; then + C_RED=$'\e[31m'; C_GRN=$'\e[32m'; C_YEL=$'\e[33m'; C_RST=$'\e[0m' + else + C_RED=""; C_GRN=""; C_YEL=""; C_RST="" + fi +} + +# Map VERSION_ID to the active tree + image. No I/O, unit-testable. +select_version_paths() { + if [[ "$VERSION_ID" == "15.0" ]]; then + ACTIVE_TREE="$SBO_TREE_STABLE"; ACTIVE_IMAGE="$IMAGE_STABLE" + else + ACTIVE_TREE="$SBO_TREE_CURRENT"; ACTIVE_IMAGE="$IMAGE_CURRENT" + fi +} + +# require_config: the external config must exist and set the version's paths. +require_config() { + if [[ ! -f "$TB_CONFIG" ]]; then + cat >&2 <<EOF +No config file: $TB_CONFIG +Copy the example and edit it: + + mkdir -p "\$(dirname "$TB_CONFIG")" + cp .extras/test-build-config.example "$TB_CONFIG" + \${EDITOR:-vi} "$TB_CONFIG" +EOF + exit 1 + fi + select_version_paths + if [[ -z "$ACTIVE_TREE" || -z "$ACTIVE_IMAGE" ]]; then + echo "Config $TB_CONFIG is missing the tree/image for version '$VERSION_ID'." >&2 + exit 1 + fi + if [[ ! -d "$ACTIVE_TREE" ]]; then + echo "SBo tree for '$VERSION_ID' does not exist: $ACTIVE_TREE" >&2 + exit 1 + fi +} + +main() { + parse_args "$@" + init_color + require_config +} + +main "$@" +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `bash .extras/test-logic.sh` +Expected: PASS (the new version-selection checks included). + +- [ ] **Step 5: Write the config example** + +Create `.extras/test-build-config.example`: + +```bash +# test-build config. Copy to ~/.config/sbo-testbuild/config and edit. +# Override the path with the TB_CONFIG environment variable. + +# Local SBo trees, one per target Slackware version. +SBO_TREE_CURRENT="/home/danix/SBo/current" +SBO_TREE_STABLE="/home/danix/SBo/15.0" + +# Ready image tags (built by a separate job, consumed here by tag). +IMAGE_CURRENT="sbo-testbuild:current" +IMAGE_STABLE="sbo-testbuild:15.0" + +# Where per-run logs land. +LOG_ROOT="/home/danix/.cache/sbo-test-build" + +# Dependency package cache (host dir). Empty disables caching. +# Namespaced internally by image digest, so it self-invalidates on image update +# and keeps current vs 15.0 separate. +PKG_CACHE="/home/danix/.cache/sbo-test-build/pkgcache" +``` + +- [ ] **Step 6: Commit** + +```bash +git add .extras/test-build .extras/test-build-config.example .extras/test-logic.sh +SBOLINT=no git commit -m 'test-build: config, arg parsing, version selection' +``` + +--- + +## Task 3: Port the SBo-tree lookup + resolver + self-check + +**Files:** +- Modify: `.extras/test-build` +- Test: `.extras/test-logic.sh` + +- [ ] **Step 1: Write the failing test** + +Append to `.extras/test-logic.sh` before the result block: + +```bash +# --- resolution ------------------------------------------------------------- +# Fake SBo tree under one category. mk <prog> "<REQUIRES>". +mk() { mkdir -p "$T/cat/$1"; echo "REQUIRES=\"$2\"" > "$T/cat/$1/$1.info"; } +mk c "" +mk b "c" +mk a "b %README%" +mk d "nonexistentpkg" +mk e "f" +mk f "e" +mk g "b" + +SBO_TREE_ROOTS=("$T") # resolver reads this global (set by select in real runs) + +resolve_target "$T/cat/a" +order=""; for x in "${RESOLVED_ORDER[@]}"; do order+="$(basename "$x") "; done +order="${order% }" +[[ "$order" == "c b a" ]] && ok "topo order c b a" || bad "topo order, got: [$order]" +[[ "${HAS_README[$T/cat/a]:-}" == "1" ]] && ok "%README% recorded" || bad "%README% not recorded" +[[ ${#UNMET[@]} -eq 0 ]] && ok "no false unmet" || bad "unexpected unmet" + +resolve_target "$T/cat/d" +[[ ${#UNMET[@]} -eq 1 ]] && ok "unmet-dep caught" || bad "unmet-dep missed" + +resolve_target "$T/cat/e" +[[ ${#CYCLES[@]} -ge 1 ]] && ok "cycle caught" || bad "cycle missed" +``` + +Note: the base-installed check (`installed_in_base`) is exercised in real runs +against the container, not here. In the self-check `BASE_DB` is empty, so a dep +that is neither in the tree nor a known override reads as UNMET (which is the +behavior we test with `d`). + +- [ ] **Step 2: Run test to verify it fails** + +Run: `bash .extras/test-logic.sh` +Expected: FAIL, `resolve_target: command not found`. + +- [ ] **Step 3: Implement lookup + resolver (ported)** + +Add to `.extras/test-build` above `main()`: + +```bash +# ============================================================================= +# SBo tree lookup +# ============================================================================= +# In this tool there is one active tree per run, but the resolver reads an array +# named SBO_TREE_ROOTS so the ported logic and its self-check match sbo-batch-test. +# main() sets SBO_TREE_ROOTS=("$ACTIVE_TREE") after require_config. +declare -a SBO_TREE_ROOTS=() + +find_slackbuild_dir() { + local prog="$1" root d + for root in "${SBO_TREE_ROOTS[@]}"; do + [[ -d "$root" ]] || continue + for d in "$root"/*/"$prog"; do + if [[ -d "$d" && -f "$d/$prog.info" ]]; then + echo "$d"; return 0 + fi + done + done + return 1 +} + +category_of() { basename "$(dirname "$1")"; } +pkg_key() { echo "$(category_of "$1")/$(basename "$1")"; } + +read_requires() { + local info="$1" + # shellcheck disable=SC1090 + ( set +u; source "$info"; echo "${REQUIRES:-}" ) +} + +version_of() { + local dir="$1" + local info="$dir/$(basename "$dir").info" + [[ -f "$info" ]] || return + local v; v="$(grep -m1 '^VERSION=' "$info" | cut -d'"' -f2)" + echo "$v" +} + +# ============================================================================= +# DEPENDENCY RESOLUTION (topo sort + cycle detection, LOCAL tree only) +# ============================================================================= +declare -a RESOLVED_ORDER=() +declare -A UNMET=() +declare -a CYCLES=() +declare -A HAS_README=() +declare -A _vstate=() + +# Is a prog already present in the container base? Overridden at build time to +# consult the image; in resolution we treat "in base" as a callback so the pure +# topo logic stays testable. Default: not in base (self-check has no base db). +installed_in_base() { return 1; } + +_resolve_visit() { + local dir="$1" parent="$2" + local key; key="$(basename "$dir")" + + if [[ "${_vstate[$dir]:-}" == "1" ]]; then return 0; fi + if [[ "${_vstate[$dir]:-}" == "0" ]]; then + CYCLES+=("cycle involving $key (pulled in via $parent)") + return 1 + fi + _vstate["$dir"]=0 + + local info="$dir/$(basename "$dir").info" + local req tok depdir rc=0 + req="$(read_requires "$info")" + for tok in $req; do + if [[ "$tok" == "%README%" ]]; then + HAS_README["$dir"]=1 + continue + fi + if depdir="$(find_slackbuild_dir "$tok")"; then + _resolve_visit "$depdir" "$key" || rc=1 + elif installed_in_base "$tok"; then + : + else + UNMET["$tok"]="needed by $key" + rc=1 + fi + done + + _vstate["$dir"]=1 + RESOLVED_ORDER+=("$dir") + return $rc +} + +resolve_target() { + local dir="$1" + RESOLVED_ORDER=() + CYCLES=() + UNMET=() + _vstate=() + _resolve_visit "$dir" "(top)" +} +``` + +Update `main()` to set the resolver's tree array after `require_config`: + +```bash +main() { + parse_args "$@" + init_color + require_config + SBO_TREE_ROOTS=("$ACTIVE_TREE") +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `bash .extras/test-logic.sh` +Expected: PASS (topo order, %README%, unmet, cycle checks green). + +- [ ] **Step 5: Commit** + +```bash +git add .extras/test-build .extras/test-logic.sh +SBOLINT=no git commit -m 'test-build: port SBo lookup + dependency resolver' +``` + +--- + +## Task 4: Override engine (drop / rename / fetch) + UNMET stop + +**Files:** +- Modify: `.extras/test-build` +- Create: `.extras/overrides.example` +- Test: `.extras/test-logic.sh` + +The override file encodes current-vs-stable deltas. It is applied ONLY when +`VERSION_ID == current` (15.0 is the SBo baseline). Rules: + +- `drop: <prog>` remove from the build order (already in -current base) +- `rename: <old> -> <new>` rewrite the dep name before it is looked up +- `fetch: <prog>` mark this dep to come from sbopkg in-container, not the local tree + +- [ ] **Step 1: Write the failing test** + +Append to `.extras/test-logic.sh` before the result block: + +```bash +# --- overrides -------------------------------------------------------------- +OV=$(mktemp) +cat > "$OV" <<'EOF' +# comment ignored +drop: dropme +rename: oldname -> newname +fetch: fetchme +EOF +TB_OVERRIDES="$OV" +load_overrides + +[[ "${OV_DROP[dropme]:-}" == "1" ]] && ok "override drop parsed" || bad "drop not parsed" +[[ "${OV_RENAME[oldname]:-}" == "newname" ]] && ok "override rename parsed" || bad "rename not parsed" +[[ "${OV_FETCH[fetchme]:-}" == "1" ]] && ok "override fetch parsed" || bad "fetch not parsed" + +# rename maps a token +[[ "$(apply_rename oldname)" == "newname" ]] && ok "apply_rename maps" || bad "apply_rename wrong: [$(apply_rename oldname)]" +[[ "$(apply_rename untouched)" == "untouched" ]] && ok "apply_rename passthrough" || bad "apply_rename mangled untouched" + +# drop filters the order (build a fake order of dirs; dropme is removed) +mk dropme "" +mk keepme "" +RESOLVED_ORDER=("$T/cat/dropme" "$T/cat/keepme") +apply_overrides_to_order +kept=""; for x in "${RESOLVED_ORDER[@]}"; do kept+="$(basename "$x") "; done +kept="${kept% }" +[[ "$kept" == "keepme" ]] && ok "drop removes from order" || bad "drop failed, order=[$kept]" +rm -f "$OV" + +# overrides only apply on current; on 15.0 they are a no-op +VERSION_ID="15.0" +TB_OVERRIDES="$OV" # file gone; must not error, must clear maps +load_overrides +[[ ${#OV_DROP[@]} -eq 0 ]] && ok "overrides inert on 15.0" || bad "overrides applied on 15.0" +VERSION_ID="current" +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `bash .extras/test-logic.sh` +Expected: FAIL, `load_overrides: command not found`. + +- [ ] **Step 3: Implement the override engine** + +Add to `.extras/test-build` above `main()`: + +```bash +# ============================================================================= +# current-vs-stable overrides. Parsed from $TB_OVERRIDES. Applied only when +# targeting -current (15.0 is the SBo baseline, no deltas). +# ============================================================================= +declare -A OV_DROP=() # prog -> 1 +declare -A OV_RENAME=() # old -> new +declare -A OV_FETCH=() # prog -> 1 + +load_overrides() { + OV_DROP=(); OV_RENAME=(); OV_FETCH=() + # 15.0 is the baseline: no overrides. + [[ "$VERSION_ID" == "15.0" ]] && return + [[ -f "$TB_OVERRIDES" ]] || return + local line kind rest + while IFS= read -r line; do + line="${line%%#*}" # strip comments + line="${line#"${line%%[![:space:]]*}"}" # ltrim + [[ -z "$line" ]] && continue + kind="${line%%:*}"; rest="${line#*:}" + kind="${kind//[[:space:]]/}" + rest="${rest#"${rest%%[![:space:]]*}"}" # ltrim value + case "$kind" in + drop) OV_DROP["${rest//[[:space:]]/}"]=1 ;; + fetch) OV_FETCH["${rest//[[:space:]]/}"]=1 ;; + rename) + # rest is "old -> new" + local old new + old="${rest%%->*}"; new="${rest##*->}" + old="${old//[[:space:]]/}"; new="${new//[[:space:]]/}" + [[ -n "$old" && -n "$new" ]] && OV_RENAME["$old"]="$new" ;; + *) echo "WARN: unknown override rule: $line" >&2 ;; + esac + done < "$TB_OVERRIDES" +} + +# Map a dep token through rename rules (identity if no rule). +apply_rename() { + local tok="$1" + echo "${OV_RENAME[$tok]:-$tok}" +} + +# Remove dropped packages from RESOLVED_ORDER in place. +apply_overrides_to_order() { + local d prog keep=() + for d in "${RESOLVED_ORDER[@]}"; do + prog="$(basename "$d")" + [[ "${OV_DROP[$prog]:-}" == "1" ]] && continue + keep+=("$d") + done + RESOLVED_ORDER=("${keep[@]:-}") +} +``` + +Wire `apply_rename` into the resolver so renamed deps look up under the new +name. In `_resolve_visit`, change the dep-lookup line: + +```bash + for tok in $req; do + if [[ "$tok" == "%README%" ]]; then + HAS_README["$dir"]=1 + continue + fi + tok="$(apply_rename "$tok")" # <-- add this line + if [[ "${OV_FETCH[$tok]:-}" == "1" ]]; then + FETCH_DEPS["$tok"]=1 # satisfied via sbopkg at build time + continue + fi + if depdir="$(find_slackbuild_dir "$tok")"; then + _resolve_visit "$depdir" "$key" || rc=1 + elif installed_in_base "$tok"; then + : + else + UNMET["$tok"]="needed by $key" + rc=1 + fi + done +``` + +Add the `FETCH_DEPS` global with the other resolver globals: + +```bash +declare -A FETCH_DEPS=() # prog -> 1: resolve via sbopkg in-container +``` + +And reset it in `resolve_target`: + +```bash +resolve_target() { + local dir="$1" + RESOLVED_ORDER=() + CYCLES=() + UNMET=() + FETCH_DEPS=() + _vstate=() + _resolve_visit "$dir" "(top)" +} +``` + +Load overrides in `main()` (before any resolve happens, after tree setup): + +```bash +main() { + parse_args "$@" + init_color + require_config + SBO_TREE_ROOTS=("$ACTIVE_TREE") + load_overrides +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `bash .extras/test-logic.sh` +Expected: PASS (all override checks green). + +- [ ] **Step 5: Write the overrides example** + +Create `.extras/overrides.example`: + +``` +# current-vs-stable dep overrides for test-build. +# Copy to ~/.config/sbo-testbuild/overrides and edit. Applied only when +# targeting -current (15.0 is the SBo baseline). One rule per line. +# +# drop: <prog> dep is in 15.0 but already in the -current base +# rename: <old> -> <new> dep was renamed on -current +# fetch: <prog> dep was removed from the -current tree; pull via sbopkg +# +# Start light: add rules only when a real package needs them. + +# drop: rust +# rename: python3-foo -> foo +# fetch: somelib +``` + +- [ ] **Step 6: Commit** + +```bash +git add .extras/test-build .extras/overrides.example .extras/test-logic.sh +SBOLINT=no git commit -m 'test-build: current-vs-stable override engine' +``` + +--- + +## Task 5: Port the dep cache, namespaced by image digest + +**Files:** +- Modify: `.extras/test-build` +- Test: `.extras/test-logic.sh` + +The cache is the ported sbo-batch-test cache with one change: the cache root +seen by `cache_*` is `$PKG_CACHE/<image-digest>`, computed once per run. A new +digest gives a fresh namespace (old deps ignored), which is the invalidation. + +- [ ] **Step 1: Write the failing test** + +Append to `.extras/test-logic.sh` before the result block: + +```bash +# --- cache ------------------------------------------------------------------ +PKG_CACHE=$(mktemp -d) +CACHE_ROOT="$PKG_CACHE/sha256-deadbeef" # simulate a resolved digest namespace +mkc() { mkdir -p "$CACHE_ROOT/$1/$2"; : > "$CACHE_ROOT/$1/$2/$3"; } + +mkc net libfoo "libfoo-1.1-x86_64-1_danix.txz" +[[ "$(cache_decision net libfoo 1.1)" == "cached" ]] && ok "cache hit on version match" || bad "cache_decision got [$(cache_decision net libfoo 1.1)]" +[[ "$(cache_decision net libfoo 1.2)" == "bump:1.1:1.2" ]] && ok "cache bump reported" || bad "cache bump got [$(cache_decision net libfoo 1.2)]" +[[ "$(cache_decision net libbar 1.0)" == "new" ]] && ok "cache new for absent" || bad "cache new got [$(cache_decision net libbar 1.0)]" + +hit="$(cache_path net libfoo 1.1)" +[[ "$hit" == "$CACHE_ROOT/net/libfoo/libfoo-1.1-x86_64-1_danix.txz" ]] && ok "cache_path returns hit" || bad "cache_path got [$hit]" +[[ -z "$(cache_path net libfoo 9.9)" ]] && ok "cache_path empty on miss" || bad "cache_path not empty on miss" + +srctmp=$(mktemp -d); : > "$srctmp/libfoo-1.2-x86_64-1_danix.txz" +cache_store net libfoo "$srctmp/libfoo-1.2-x86_64-1_danix.txz" +count=$(find "$CACHE_ROOT/net/libfoo" -name '*.t?z' | wc -l) +[[ "$count" -eq 1 ]] && ok "cache_store evicts to one file" || bad "cache_store left $count files" +[[ -e "$CACHE_ROOT/net/libfoo/libfoo-1.2-x86_64-1_danix.txz" ]] && ok "cache_store stored new file" || bad "cache_store did not store" +rm -rf "$srctmp" + +# disabled cache -> new +CACHE_ROOT_SAVE="$CACHE_ROOT"; USE_CACHE=0 +[[ "$(cache_decision net libfoo 1.2)" == "new" ]] && ok "--no-cache disables (new)" || bad "disabled cache got [$(cache_decision net libfoo 1.2)]" +USE_CACHE=1; CACHE_ROOT="$CACHE_ROOT_SAVE" +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `bash .extras/test-logic.sh` +Expected: FAIL, `cache_decision: command not found`. + +- [ ] **Step 3: Implement the cache (ported, keyed on CACHE_ROOT + USE_CACHE)** + +Add to `.extras/test-build` above `main()`: + +```bash +# ============================================================================= +# Dependency cache. Layout: $CACHE_ROOT/<cat>/<prog>/<prog>-<ver>-...txz where +# CACHE_ROOT = $PKG_CACHE/<image-digest> (set per run by resolve_cache_root). +# Key is prog+version. --no-cache (USE_CACHE=0) or empty PKG_CACHE disables. +# ============================================================================= +CACHE_ROOT="" # set by resolve_cache_root once the image digest is known + +# True when the cache is usable this run. +_cache_on() { [[ $USE_CACHE -eq 1 && -n "$PKG_CACHE" && -n "$CACHE_ROOT" ]]; } + +_cache_ver_of() { + local prog="$1" base="$2" + base="${base#"$prog"-}" + echo "${base%%-*}" +} + +# cache_decision <cat> <prog> <version> -> cached | bump:OLD:NEW | new +cache_decision() { + local cat="$1" prog="$2" version="$3" + _cache_on || { echo new; return; } + local dir="$CACHE_ROOT/$cat/$prog" + local f newest="" + for f in "$dir/$prog"-*.t?z; do + [[ -e "$f" ]] || continue + [[ -z "$newest" || "$f" -nt "$newest" ]] && newest="$f" + done + [[ -z "$newest" ]] && { echo new; return; } + local have; have="$(_cache_ver_of "$prog" "$(basename "$newest")")" + if [[ "$have" == "$version" ]]; then echo cached; else echo "bump:$have:$version"; fi +} + +cache_path() { + local cat="$1" prog="$2" version="$3" + _cache_on || return + local dir="$CACHE_ROOT/$cat/$prog" + local f newest="" + for f in "$dir/$prog"-*.t?z; do + [[ -e "$f" ]] || continue + [[ -z "$newest" || "$f" -nt "$newest" ]] && newest="$f" + done + [[ -z "$newest" ]] && return + [[ "$(_cache_ver_of "$prog" "$(basename "$newest")")" == "$version" ]] && echo "$newest" +} + +cache_store() { + local cat="$1" prog="$2" src="$3" + _cache_on || return + local dir="$CACHE_ROOT/$cat/$prog" + mkdir -p "$dir" + rm -f "$dir"/*.t?z + cp -a "$src" "$dir/" +} + +cache_label() { + local dir="$1" is_target="$2" + local cat prog ver dec + cat="$(category_of "$dir")"; prog="$(basename "$dir")"; ver="$(version_of "$dir")" + dec="$(cache_decision "$cat" "$prog" "$ver")" + local label + case "$dec" in + cached) label="cached ($ver)" ;; + bump:*) label="rebuild: ${dec#bump:}"; label="${label/:/ -> }" ;; + *) label="build (new)" ;; + esac + if [[ "$is_target" == "1" ]]; then + case "$dec" in + cached) label="build (cached $ver, rebuilt as target)" ;; + esac + echo "target, $label" + else + echo "$label" + fi +} + +# Compute CACHE_ROOT from the image's digest. Falls back to the tag if the +# digest cannot be read (still isolates per image reference). +resolve_cache_root() { + [[ -z "$PKG_CACHE" ]] && { CACHE_ROOT=""; return; } + local digest + digest="$(docker image inspect --format '{{index .Id}}' "$ACTIVE_IMAGE" 2>/dev/null)" + [[ -z "$digest" ]] && digest="tag-${ACTIVE_IMAGE//[^a-zA-Z0-9._-]/_}" + digest="${digest//[^a-zA-Z0-9._-]/_}" + CACHE_ROOT="$PKG_CACHE/$digest" + mkdir -p "$CACHE_ROOT" +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `bash .extras/test-logic.sh` +Expected: PASS (all cache checks green). + +- [ ] **Step 5: Commit** + +```bash +git add .extras/test-build .extras/test-logic.sh +SBOLINT=no git commit -m 'test-build: port dep cache, namespaced by image digest' +``` + +--- + +## Task 6: BLOCKED-BY-DEP propagation (ported) + +**Files:** +- Modify: `.extras/test-build` +- Test: `.extras/test-logic.sh` + +- [ ] **Step 1: Write the failing test** + +Append to `.extras/test-logic.sh` before the result block: + +```bash +# --- BLOCKED-BY-DEP (depends_on_failed) ------------------------------------- +dead=(b) +if depends_on_failed "$T/cat/g" dead; then ok "g blocked when b dead"; else bad "g should block on b"; fi +if depends_on_failed "$T/cat/a" dead; then ok "a blocked when b dead (direct)"; else bad "a should block on b"; fi + +dead=(c) +if depends_on_failed "$T/cat/a" dead; then bad "a wrongly blocked on c"; else ok "a not directly blocked by c"; fi +if depends_on_failed "$T/cat/b" dead; then ok "b blocked when c dead"; else bad "b should block on c"; fi + +dead=() +if depends_on_failed "$T/cat/a" dead; then bad "a blocked with empty dead"; else ok "no block when nothing dead"; fi + +dead=("%README%") +if depends_on_failed "$T/cat/a" dead; then bad "%README% treated as dep"; else ok "%README% not treated as dep"; fi +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `bash .extras/test-logic.sh` +Expected: FAIL, `depends_on_failed: command not found`. + +- [ ] **Step 3: Implement depends_on_failed (ported)** + +Add to `.extras/test-build` above `main()`: + +```bash +# Does SlackBuild dir $1 directly require any prog in the dead list (nameref $2)? +# Direct-requires check only; transitive blocking works because run_target +# iterates in topo order, propagating a failure one hop per package. +depends_on_failed() { + local dir="$1"; local -n failed="$2" + local info="$dir/$(basename "$dir").info" + local req tok f + req="$(read_requires "$info")" + for tok in $req; do + [[ "$tok" == "%README%" ]] && continue + for f in "${failed[@]:-}"; do + [[ "$tok" == "$f" ]] && return 0 + done + done + return 1 +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `bash .extras/test-logic.sh` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add .extras/test-build .extras/test-logic.sh +SBOLINT=no git commit -m 'test-build: port BLOCKED-BY-DEP propagation' +``` + +--- + +## Task 7: The container build step (build_one via docker) + +**Files:** +- Modify: `.extras/test-build` + +No self-check here (docker/build is out of the pure-logic reach, same boundary +as sbo-batch-test). Verified by a real run in Task 10. + +- [ ] **Step 1: Implement build_one + lint_pkg** + +Add to `.extras/test-build` above `main()`. + +`lint_pkg` is ported verbatim: + +```bash +# lint_pkg <txz> <logf> -> run sbopkglint on a built package. Host-side, target +# only, fail-soft (skip if absent, never change SUCCESS). +lint_pkg() { + local txz="$1" logf="$2" + if ! command -v sbopkglint >/dev/null 2>&1; then + echo " sbopkglint not installed on host, skipping lint" + return 0 + fi + echo " sbopkglint $(basename "$txz") ..." + echo "===== sbopkglint: $(basename "$txz") =====" >> "$logf" + local out rc + out="$(sbopkglint "$txz" 2>&1)"; rc=$? + printf '%s\n' "$out" >> "$logf" + if [[ $rc -eq 0 ]]; then + echo " lint: ${C_GRN}clean${C_RST}" + else + echo " lint: ${C_RED}findings${C_RST} (see $(basename "$logf")):" + printf '%s\n' "$out" | sed 's/^/ /' + fi +} +``` + +`build_one` runs the same download/md5/build/installpkg steps as sbo-batch-test, +but inside a container via `docker run` instead of `chroot`. The container gets: +the SlackBuild dir mounted read-only, a work dir for output, and (for deps) the +cached .txz mounted in. `sbopkglint` runs host-side on the copied-out package. + +```bash +# build_one <slackbuild-dir> <is_target 0|1> [container-name] +# Runs the build in a throwaway container. Sets ST_STATUS/ST_REASON/ST_TIME. +# Successful builds copy their package to a host workdir; deps are cached and +# the target is linted. Returns 0 on SUCCESS/CACHED, 1 otherwise. +build_one() { + local dir="$1" is_target="${2:-0}" + local prog cat key + prog="$(basename "$dir")"; cat="$(category_of "$dir")"; key="$cat/$prog" + local logf="$RUN_DIR/${cat}_${prog}.log" + local start; start=$(date +%s) + local version; version="$(version_of "$dir")" + + [[ "${HAS_README[$dir]:-}" == "1" ]] && ST_README["$key"]=1 + + # Dep with a version-matching cached package: installpkg it into the shared + # dep-package dir; no build. The target never takes this path. + if [[ "$is_target" != "1" ]]; then + local cached; cached="$(cache_path "$cat" "$prog" "$version")" + if [[ -n "$cached" ]]; then + cp -a "$cached" "$DEPS_DIR/" + { + echo "===== test-build: $prog (from cache) =====" + echo "cached package: $(basename "$cached")" + } >> "$logf" + ST_TIME["$key"]=$(( $(date +%s) - start )) + ST_STATUS["$key"]="CACHED" + return 0 + fi + fi + + # Build in a container. Mounts: + # $dir -> /sbo/pkg (ro, the SlackBuild) + # $DEPS_DIR -> /sbo/deps (rw, already-built dep .txz to installpkg first) + # $BUILD_OUT -> /sbo/out (rw, where the built package is copied out) + # The in-container script installs any deps present, then builds the target, + # writes a status token to /sbo/out/$prog.status, and copies the package out. + local statf="$BUILD_OUT/$prog.status" + rm -f "$statf" + + # fetch deps (removed from -current tree): let the container's sbopkg build + # them first. FETCH_DEPS is the set collected during resolution. + local fetch_list="" + local fp + for fp in "${!FETCH_DEPS[@]}"; do fetch_list+="$fp "; done + + docker run --rm \ + -v "$dir":/sbo/pkg:ro \ + -v "$DEPS_DIR":/sbo/deps \ + -v "$BUILD_OUT":/sbo/out \ + -e PROG="$prog" \ + -e FETCH_LIST="$fetch_list" \ + "$ACTIVE_IMAGE" /bin/bash -s >>"$logf" 2>&1 <<'CONTAINER_EOF' +set -uo pipefail +prog="$PROG" +statf="/sbo/out/$prog.status" + +# 0. install already-built dependency packages (order guaranteed by the host). +for d in /sbo/deps/*.t?z; do + [[ -e "$d" ]] || continue + installpkg --terse "$d" || { echo "INSTALL-FAILED (dep $d)"; echo INSTALL-FAILED > "$statf"; exit 1; } +done + +# 0b. fetch-from-SBo deps via sbopkg (removed from the -current tree). +for f in $FETCH_LIST; do + echo "sbopkg-building fetch dep: $f" + sbopkg -B -i "$f" || { echo "BUILD-FAILED (fetch dep $f)"; echo BUILD-FAILED > "$statf"; exit 1; } +done + +# copy the SlackBuild out of the read-only mount so it can write there. +cp -a /sbo/pkg /sbo/build +cd /sbo/build || { echo BUILD-FAILED > "$statf"; exit 1; } +. ./"$prog".info + +export OUTPUT=/sbo/out +mkdir -p "$OUTPUT" + +echo "===== test-build: $prog =====" +echo "PRGNAM=${PRGNAM:-$prog} VERSION=${VERSION:-?} BUILD=${BUILD:-?} TAG=${TAG:-?}" +echo "uname -m: $(uname -m) OUTPUT=$OUTPUT" +echo "REQUIRES=${REQUIRES:-}" +echo "=================================" + +if [ "$(uname -m)" = "x86_64" ] && [ -n "${DOWNLOAD_x86_64:-}" ] && [ "${DOWNLOAD_x86_64}" != "UNSUPPORTED" ] && [ "${DOWNLOAD_x86_64}" != "UNTESTED" ]; then + DL="$DOWNLOAD_x86_64"; MD="$MD5SUM_x86_64" +else + DL="$DOWNLOAD"; MD="$MD5SUM" +fi + +for u in $DL; do + wget -c --tries=3 "$u" || { echo DOWNLOAD-FAILED > "$statf"; exit 1; } +done + +set -- $MD +for u in $DL; do + f="$(basename "$u")" + want="$1"; shift + got="$(md5sum "$f" | cut -d' ' -f1)" + if [ "$got" != "$want" ]; then + echo "MD5 mismatch on $f: want $want got $got" + echo MD5-MISMATCH > "$statf"; exit 1 + fi +done + +chmod +x ./"$prog".SlackBuild +if ! ./"$prog".SlackBuild; then + echo BUILD-FAILED > "$statf"; exit 1 +fi + +pkg="$(ls -t "$OUTPUT"/"$prog"-*.t?z 2>/dev/null | head -n1)" +if [ -z "$pkg" ]; then + echo "No package produced in $OUTPUT" + echo BUILD-FAILED > "$statf"; exit 1 +fi +if ! installpkg --terse "$pkg"; then + echo INSTALL-FAILED > "$statf"; exit 1 +fi +echo "===== installed files: $(basename "$pkg") =====" +pkgname="$(basename "$pkg")"; pkgname="${pkgname%.t?z}" +cat "/var/log/packages/$pkgname" 2>/dev/null || echo "(package db entry not found)" +echo "=================================" +echo SUCCESS > "$statf" +CONTAINER_EOF + + local status="BUILD-FAILED" + [[ -f "$statf" ]] && status="$(cat "$statf")" + ST_TIME["$key"]=$(( $(date +%s) - start )) + ST_STATUS["$key"]="$status" + + if [[ "$status" == "SUCCESS" ]]; then + # locate the built package copied to the host workdir + local built newest="" + for built in "$BUILD_OUT/${prog}"-*.t?z; do + [[ -e "$built" ]] || continue + [[ -z "$newest" || "$built" -nt "$newest" ]] && newest="$built" + done + if [[ -n "$newest" ]]; then + if [[ "$is_target" != "1" ]]; then + cache_store "$cat" "$prog" "$newest" + # make the dep available to later builds in this run + cp -a "$newest" "$DEPS_DIR/" + else + lint_pkg "$newest" "$logf" + fi + fi + return 0 + fi + ST_REASON["$key"]="see $(basename "$logf")" + return 1 +} +``` + +- [ ] **Step 2: Sanity-check bash syntax** + +Run: `bash -n .extras/test-build` +Expected: no output (syntax OK). + +- [ ] **Step 3: Run the self-check still passes (no regression)** + +Run: `bash .extras/test-logic.sh` +Expected: PASS (build_one is not exercised, but sourcing must not break). + +- [ ] **Step 4: Commit** + +```bash +git add .extras/test-build +SBOLINT=no git commit -m 'test-build: container build step (build_one via docker)' +``` + +--- + +## Task 8: run_target orchestration + summary + +**Files:** +- Modify: `.extras/test-build` + +- [ ] **Step 1: Implement run_target + print_summary + confirm** + +Add to `.extras/test-build` above `main()`. `print_summary` is ported verbatim +(status set is the same). `run_target` replaces the overlay lifecycle with the +per-run docker workdirs and adds the printed-order + Y/n confirm. + +```bash +# confirm_order: print the resolved order (overrides marked) and ask to proceed. +# --yes skips the prompt but the order is still printed. --dry-run never reaches +# here. Returns 0 to proceed, 1 to abort. +confirm_order() { + local target_dir="$1" + echo " build order (${VERSION_ID}):" + local d + for d in "${RESOLVED_ORDER[@]}"; do + local it=0; [[ "$d" == "$target_dir" ]] && it=1 + local rdm=""; [[ "${HAS_README[$d]:-}" == 1 ]] && rdm=" [%README%]" + printf " %-30s %s%s\n" "$(pkg_key "$d")" "$(cache_label "$d" "$it")" "$rdm" + echo "$(pkg_key "$d")" >> "$RUN_DIR/build-order.txt" + done + # note fetch deps (built via sbopkg in-container, not in the order list) + local fp + for fp in "${!FETCH_DEPS[@]}"; do + printf " %-30s %s\n" "$fp" "fetch (sbopkg in container)" + done + [[ $ASSUME_YES -eq 1 ]] && return 0 + local reply + read -rp " Proceed? [Y/n] " reply + [[ -z "$reply" || "$reply" =~ ^[Yy]$ ]] +} + +# run_target <target-slackbuild-dir> +run_target() { + local target_dir="$1" + local tkey; tkey="$(pkg_key "$target_dir")" + + echo + echo "=== Target: $tkey (${VERSION_ID}) ===" + resolve_target "$target_dir" + apply_overrides_to_order + + # Hard resolution failures: report and stop, do not build. + if [[ ${#CYCLES[@]} -gt 0 || ${#UNMET[@]} -gt 0 ]]; then + local why="" + if [[ ${#UNMET[@]} -gt 0 ]]; then + local u + for u in "${!UNMET[@]}"; do why+="unmet:$u(${UNMET[$u]}) "; done + fi + [[ ${#CYCLES[@]} -gt 0 ]] && why+="${CYCLES[*]}" + ST_STATUS["$tkey"]="UNMET-DEP" + ST_REASON["$tkey"]="$why" + echo " resolution failed: $why" + echo " add an override rule ($TB_OVERRIDES) and rerun, or fix the tree." >&2 + return + fi + + if [[ $DRY_RUN -eq 1 ]]; then + echo " build order (dry-run):" + local d + for d in "${RESOLVED_ORDER[@]}"; do + local it=0; [[ "$d" == "$target_dir" ]] && it=1 + local rdm=""; [[ "${HAS_README[$d]:-}" == 1 ]] && rdm=" [%README%]" + printf " %-30s %s%s\n" "$(pkg_key "$d")" "$(cache_label "$d" "$it")" "$rdm" + echo "$(pkg_key "$d")" >> "$RUN_DIR/build-order.txt" + done + local fp + for fp in "${!FETCH_DEPS[@]}"; do + printf " %-30s %s\n" "$fp" "fetch (sbopkg in container)" + done + return + fi + + if ! confirm_order "$target_dir"; then + echo " aborted." + ST_STATUS["$tkey"]="ABORTED" + return + fi + + # Per-run docker workdirs (host side, discarded after the run). + DEPS_DIR="$RUN_DIR/deps"; BUILD_OUT="$RUN_DIR/out" + mkdir -p "$DEPS_DIR" "$BUILD_OUT" + + local d failed_progs=() + for d in "${RESOLVED_ORDER[@]}"; do + local key; key="$(pkg_key "$d")" + local prog; prog="$(basename "$d")" + + if depends_on_failed "$d" failed_progs; then + ST_STATUS["$key"]="BLOCKED-BY-DEP" + ST_REASON["$key"]="blocked by failed dep" + [[ "${HAS_README[$d]:-}" == "1" ]] && ST_README["$key"]=1 + echo " $key: BLOCKED-BY-DEP" + failed_progs+=("$prog") + continue + fi + + local it=0; [[ "$d" == "$target_dir" ]] && it=1 + echo " building $key ..." + if build_one "$d" "$it"; then + echo " $key: ${ST_STATUS[$key]} (${ST_TIME[$key]}s)" + else + echo " $key: ${ST_STATUS[$key]} (${ST_TIME[$key]}s)" + failed_progs+=("$prog") + fi + done +} + +# ============================================================================= +# SUMMARY (ported) +# ============================================================================= +print_summary() { + local total=$SECONDS + local succ=0 fail=0 blocked=0 cached=0 + local summary="$RUN_DIR/summary.log" + + { + echo "test-build run summary" + echo "target: $TARGET_ARG version: $VERSION_ID" + echo + } > "$summary" + + echo + echo "================ SUMMARY ================" + local key + for key in "${!ST_STATUS[@]}"; do + local st="${ST_STATUS[$key]}" rsn="${ST_REASON[$key]:-}" t="${ST_TIME[$key]:-0}" + local rd=""; [[ "${ST_README[$key]:-}" == "1" ]] && rd=" [%README%]" + local col="$C_YEL" + case "$st" in + SUCCESS) col="$C_GRN"; ((succ++)) ;; + CACHED) col="$C_GRN"; ((cached++)) ;; + BLOCKED-BY-DEP|UNMET-DEP|ABORTED) col="$C_YEL"; ((blocked++)) ;; + *) col="$C_RED"; ((fail++)) ;; + esac + printf "%s%-30s %-16s%s %s%s (%ss)\n" "$col" "$key" "$st" "$C_RST" "$rsn" "$rd" "$t" + printf "%-30s %-16s %s%s (%ss)\n" "$key" "$st" "$rsn" "$rd" "$t" >> "$summary" + done + echo "----------------------------------------" + printf "%s%d succeeded%s, %s%d failed%s, %s%d blocked%s, %s%d cached%s, total %ss\n" \ + "$C_GRN" "$succ" "$C_RST" "$C_RED" "$fail" "$C_RST" "$C_YEL" "$blocked" "$C_RST" \ + "$C_GRN" "$cached" "$C_RST" "$total" + echo "logs: $RUN_DIR" + if [[ $fail -eq 0 && $blocked -eq 0 ]]; then + echo "${C_GRN}All green.${C_RST} Safe to build the SBo submission tarball on the host." + fi + { + echo + echo "$succ succeeded, $fail failed, $blocked blocked, $cached cached, total ${total}s" + echo "logs: $RUN_DIR" + } >> "$summary" +} +``` + +Add the two workdir globals near the other globals (top of the flags block): + +```bash +DEPS_DIR="" # per-run host dir of built dep .txz, mounted into the container +BUILD_OUT="" # per-run host dir where built packages are copied out +``` + +- [ ] **Step 2: Sanity-check syntax** + +Run: `bash -n .extras/test-build` +Expected: no output. + +- [ ] **Step 3: Self-check no regression** + +Run: `bash .extras/test-logic.sh` +Expected: PASS. + +- [ ] **Step 4: Commit** + +```bash +git add .extras/test-build +SBOLINT=no git commit -m 'test-build: run_target orchestration + summary + confirm' +``` + +--- + +## Task 9: Wire main(), image presence check, target lookup + +**Files:** +- Modify: `.extras/test-build` + +- [ ] **Step 1: Implement the full main()** + +Replace the current `main()` in `.extras/test-build` with: + +```bash +# Verify the selected image exists locally. It is built by a separate job; this +# script only consumes it. (Later: a local-registry docker pull slots in here.) +require_image() { + if ! docker image inspect "$ACTIVE_IMAGE" >/dev/null 2>&1; then + cat >&2 <<EOF +Image not found locally: $ACTIVE_IMAGE +It is produced by the separate image-builder job (full Slackware $VERSION_ID + +sbo-maintainer-tools + sbopkg), stored on the NAS / local registry. Load or +pull it, then rerun. This script does not build images. +EOF + exit 1 + fi +} + +main() { + parse_args "$@" + init_color + require_config + SBO_TREE_ROOTS=("$ACTIVE_TREE") + load_overrides + + # image + cache namespace (skip the image check on a pure dry-run so the order + # can be inspected without the image present). + if [[ $DRY_RUN -eq 0 ]]; then + command -v docker >/dev/null 2>&1 || { echo "docker not found in PATH." >&2; exit 1; } + require_image + fi + resolve_cache_root + + RUN_DIR="$LOG_ROOT/$(date +%Y-%m-%d_%H-%M-%S)" + mkdir -p "$RUN_DIR" + : > "$RUN_DIR/build-order.txt" + + # Single-package mode: resolve the name in the active tree. + local tdir + if ! tdir="$(find_slackbuild_dir "$TARGET_ARG")"; then + echo "Program not found in the SBo tree ($ACTIVE_TREE): $TARGET_ARG" >&2 + exit 1 + fi + + run_target "$tdir" + print_summary +} +``` + +- [ ] **Step 2: Sanity-check syntax** + +Run: `bash -n .extras/test-build` +Expected: no output. + +- [ ] **Step 3: Dry-run smoke test against the real tree (no docker needed)** + +This needs a real config. If `~/.config/sbo-testbuild/config` is not set up, +create a throwaway one pointing at an existing local SBo tree, then: + +Run: `.extras/test-build --dry-run <some-pkg-in-that-tree>` +Expected: prints `=== Target: ... ===`, a build order, exits 0, no docker call. + +- [ ] **Step 4: Self-check no regression** + +Run: `bash .extras/test-logic.sh` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add .extras/test-build +SBOLINT=no git commit -m 'test-build: wire main, image-presence check, target lookup' +``` + +--- + +## Task 10: Real end-to-end build (user-run) + README + +**Files:** +- Modify: `.extras/test-build` (only if the real run surfaces a bug) +- Create: `.extras/README.test-build` (short usage note) + +This task needs a real image and the buildsystem. Per repo rules the throwaway +docker build is Claude-OK, but the IMAGE must exist first (separate job, not yet +built). So this task is: document usage, and record the end-to-end run as a +manual verification step the user performs once an image is available. + +- [ ] **Step 1: Write the usage README** + +Create `.extras/README.test-build`: + +```markdown +# test-build + +Verify an already-published SBo package still builds on a target Slackware +version inside a throwaway docker container. + +## Setup + + mkdir -p ~/.config/sbo-testbuild + cp .extras/test-build-config.example ~/.config/sbo-testbuild/config + cp .extras/overrides.example ~/.config/sbo-testbuild/overrides + $EDITOR ~/.config/sbo-testbuild/config + +Set the two SBo tree paths, the two image tags, LOG_ROOT, and PKG_CACHE. + +The images (`sbo-testbuild:current` / `:15.0`) are FULL Slackware installs plus +`sbo-maintainer-tools` and `sbopkg`, built by a separate job (not this script). +The script errors if the tagged image is not present locally. + +## Usage + + .extras/test-build <pkg> # build on -current (default) + .extras/test-build --stable <pkg> # build on 15.0 + .extras/test-build --dry-run <pkg> # show the build order, no build + .extras/test-build --yes <pkg> # skip the confirm (order still shown) + .extras/test-build --no-cache <pkg> # rebuild all deps this run + +A green run means it is safe to build the SBo submission tarball on the host. + +## current-vs-stable overrides + +`~/.config/sbo-testbuild/overrides` (see `overrides.example`) encodes deltas +that only apply when building on -current: + + drop: <prog> already in the -current base + rename: <old> -> <new> renamed on -current + fetch: <prog> removed from the -current tree; built via sbopkg + +Start light; add rules only when a real package needs them. + +## Self-check + + bash .extras/test-logic.sh +``` + +- [ ] **Step 2: Commit the docs** + +```bash +git add .extras/README.test-build +SBOLINT=no git commit -m 'test-build: usage README' +``` + +- [ ] **Step 3: Manual end-to-end verification (user, once an image exists)** + +Documented for the user to run when a `sbo-testbuild:*` image is available: + +```bash +# pick a small already-published package with a dep, e.g. one already in the repo +.extras/test-build --yes <pkg> +``` + +Expected: build order printed, deps built (or CACHED on a second run), target +built, `sbopkglint` result shown, summary green, `All green.` line printed. +A second run of the same package should show its deps as `CACHED`. + +Record the outcome. If a real bug surfaces (mount path, sbopkg invocation, +package glob), fix it in `.extras/test-build` and re-run; add a self-check case +if the bug was in pure logic. + +--- + +## Self-Review + +Run this checklist against the spec after the plan is written: + +**1. Spec coverage:** +- purpose / one-target-per-run: Task 2 (`parse_args`), Task 9 (`main`) - covered +- two versions, current default: Task 2 (`--stable`, `select_version_paths`) - covered +- resolve local tree + topo: Task 3 - covered +- overrides drop/rename/fetch + UNMET stop: Task 4, stop in Task 8 `run_target` - covered +- external config + overrides file: Task 2, Task 4 - covered +- external image, presence check, no bake: Task 9 `require_image` - covered +- container build (download/md5/build/installpkg): Task 7 - covered +- fetch via sbopkg: Task 7 (`FETCH_LIST` loop) - covered +- dep cache, digest-namespaced, --no-cache, target always fresh: Task 5, wired Task 7/8 - covered +- target lint: Task 7 `lint_pkg`, called in Task 7 build_one - covered +- confirm order + --yes still prints: Task 8 `confirm_order` - covered +- --dry-run: Task 8 `run_target`, Task 9 skips image check - covered +- report + logs (per-pkg, summary, build-order): Task 8 `print_summary`, Task 7 logs - covered +- status values incl CACHED: Task 8 `print_summary` - covered +- self-check parity: Tasks 1-6 - covered + +**2. Placeholder scan:** no TBD/TODO in code steps; every code step shows full +code. The one deferred item (real end-to-end run) is explicit and gated on an +external image, not a placeholder. + +**3. Type consistency:** function + global names match across tasks: +`RESOLVED_ORDER`, `FETCH_DEPS`, `OV_DROP/OV_RENAME/OV_FETCH`, `CACHE_ROOT`, +`ACTIVE_TREE/ACTIVE_IMAGE`, `DEPS_DIR/BUILD_OUT`, `cache_decision/cache_path/ +cache_store/cache_label`, `select_version_paths`, `apply_overrides_to_order`, +`apply_rename`, `load_overrides`, `resolve_cache_root`, `require_image`, +`build_one(dir,is_target)`, `run_target`, `confirm_order`, `print_summary`. +Consistent. + +## Notes on deliberate simplifications + +- Single-package mode only (no category-folder / all / queue modes). This repo + bumps one package at a time. Extension point is `main`'s target lookup. +- `-j` parallelism dropped entirely (was a no-op stub upstream). Builds serial. +- fetch deps are not topo-ordered among themselves; sbopkg resolves their own + deps. Start light per the spec; revisit if a fetch dep needs another fetch dep. +``` + diff --git a/docs/plans/2026-07-13-image-builder.md b/docs/plans/2026-07-13-image-builder.md new file mode 100644 index 0000000..0c61051 --- /dev/null +++ b/docs/plans/2026-07-13-image-builder.md @@ -0,0 +1,790 @@ +# sbo-testbuild image builder — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Build the three chained bash scripts that produce `sbo-testbuild:current` / `sbo-testbuild:15.0` docker images from NAS Slackware trees and push them to a LAN registry, consumed by `.extras/test-build`. + +**Architecture:** Adapt the forge `slackware/docker-images` scripts (`bootstrap.sh`, `build-full-image.sh`) into `.extras/image-builder/`. Add a third `build-sbo-testbuild.sh` layer. Share config + helpers via `config` and `lib.sh`. Pure-logic helpers are unit-tested with an assert-based self-check; the docker chain is verified manually on the VM. + +**Tech Stack:** bash, docker, slackpkg/installpkg (Slackware), `registry:2`. + +**Reference:** design spec at `.extras/docs/specs/2026-07-13-image-builder-design.md`. Forge originals cloned at (scratchpad) `docker-images/scripts/` — re-clone with `git clone slackware_forge:slackware/docker-images.git` if gone. + +--- + +## File Structure + +``` +.extras/image-builder/ +├── config REGISTRY, MIRROR, VARIANTS, PKGDIR (sourced by all) +├── lib.sh _log/_warn/_err, fetch(), txz_hash(), require_mount() +├── bootstrap.sh base image FROM scratch (root; installpkg from NAS) +├── build-full-image.sh full image FROM base (slackpkg all series) +├── build-sbo-testbuild.sh testbuild image FROM full (installpkg sbopkg + tools) +├── test-image-builder.sh assert-based self-check for lib.sh logic +└── README VM-side setup checklist +``` + +Responsibilities: +- **config** — the only place hostnames/paths/variants live. No script hardcodes them. +- **lib.sh** — all shared, testable logic. Sourced by the three build scripts and by the test. No side effects at source time (only function defs + nothing else). +- **bootstrap/full/testbuild** — one image each, thin: guard, gate, generate Dockerfile, build, push. +- **test-image-builder.sh** — sources lib.sh, exercises fetch/txz_hash/require_mount with asserts. No docker. + +--- + +## Task 1: config + lib.sh skeleton + +**Files:** +- Create: `.extras/image-builder/config` +- Create: `.extras/image-builder/lib.sh` + +- [ ] **Step 1: Write config** + +Create `.extras/image-builder/config`: + +```sh +# Shared configuration for the sbo-testbuild image builder. +# Sourced by bootstrap.sh, build-full-image.sh, build-sbo-testbuild.sh, tests. + +REGISTRY="docker.noland.dnx:5000" +MIRROR="file:///mnt/nas" +VARIANTS=(current 15.0) # x86_64 only for now +PKGDIR="/opt/sbo-testbuild/pkgs" # sbopkg + sbo-maintainer-tools .txz +HASH_DIR="/var/cache/sbo-testbuild" # ChangeLog hashes for rebuild gating +``` + +- [ ] **Step 2: Write lib.sh with logging + require_mount** + +Create `.extras/image-builder/lib.sh`: + +```sh +# Shared helpers for the image builder. Source, do not execute. +# Sourcing must have no side effects beyond defining functions. + +_log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] [${LOG_TAG:-image-builder}] $*"; } +_warn() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] [${LOG_TAG:-image-builder}] WARNING: $*" >&2; } +_err() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] [${LOG_TAG:-image-builder}] ERROR: $*" >&2; exit 1; } + +# mirror_path MIRROR RELPATH +# Join a MIRROR (file:///... or http(s)://...) with a relative path. +mirror_path() { + local mirror="$1" rel="$2" + printf '%s/%s' "${mirror%/}" "${rel#/}" +} + +# is_local_mirror MIRROR -> 0 if file:// or bare absolute path, else 1 +is_local_mirror() { + case "$1" in + file://*) return 0 ;; + http://*|https://*) return 1 ;; + /*) return 0 ;; + *) return 1 ;; + esac +} + +# strip_scheme URL -> filesystem path for a file:// or bare-local URL +strip_scheme() { + local u="$1" + case "$u" in + file://*) printf '%s' "${u#file://}" ;; + *) printf '%s' "$u" ;; + esac +} + +# fetch SRC DEST +# Copy SRC (a full mirror URL) to DEST. Local mirror -> cp, http -> curl. +# Returns non-zero on failure; caller decides whether that is fatal. +fetch() { + local src="$1" dest="$2" + if is_local_mirror "$src"; then + local path; path="$(strip_scheme "$src")" + [[ -f "$path" ]] || return 1 + cp "$path" "$dest" + else + curl -sSfL --connect-timeout 60 "$src" -o "$dest" + fi +} + +# txz_hash DIR +# Stable hash of the set of .txz files in DIR (names + contents), order +# independent. Empty/absent dir -> empty string. +txz_hash() { + local dir="$1" + [[ -d "$dir" ]] || { printf ''; return 0; } + local files; files=$(find "$dir" -maxdepth 1 -name '*.txz' -type f | sort) + [[ -n "$files" ]] || { printf ''; return 0; } + # Hash sorted "name sha256" lines so reordering does not change output. + while IFS= read -r f; do + printf '%s %s\n' "$(basename "$f")" "$(sha256sum "$f" | cut -d' ' -f1)" + done <<< "$files" | sha256sum | cut -d' ' -f1 +} + +# require_mount VERSION +# Assert the NAS tree for VERSION is mounted (file:// mirror only). +# x86_64 tree dir is slackware64-${VERSION}. +require_mount() { + local version="$1" + is_local_mirror "$MIRROR" || return 0 # http mirror: nothing to check + local base; base="$(strip_scheme "$MIRROR")" + local dir="${base}/slackware64-${version}" + [[ -d "$dir" ]] || _err "NAS tree not mounted: ${dir}" +} +``` + +- [ ] **Step 3: Commit** + +```bash +git add .extras/image-builder/config .extras/image-builder/lib.sh +git commit -m 'image-builder: config + lib.sh helpers' +``` + +--- + +## Task 2: lib.sh self-check + +**Files:** +- Create: `.extras/image-builder/test-image-builder.sh` + +- [ ] **Step 1: Write the test** + +Create `.extras/image-builder/test-image-builder.sh`: + +```sh +#!/bin/bash +# Assert-based self-check for lib.sh pure logic. No docker, no network. +# Run: bash .extras/image-builder/test-image-builder.sh +set -u +cd "$(dirname "$0")" +source ./lib.sh + +pass=0 fail=0 +check() { # DESC EXPECTED ACTUAL + if [[ "$2" == "$3" ]]; then pass=$((pass+1)); + else fail=$((fail+1)); echo "FAIL: $1"; echo " expected: [$2]"; echo " actual: [$3]"; fi +} +check_rc() { # DESC EXPECTED_RC ACTUAL_RC + if [[ "$2" == "$3" ]]; then pass=$((pass+1)); + else fail=$((fail+1)); echo "FAIL: $1 (rc expected $2 got $3)"; fi +} + +tmp="$(mktemp -d)" +trap 'rm -rf "$tmp"' EXIT + +# --- mirror_path --- +check "mirror_path trailing slash" \ + "file:///mnt/nas/slackware64-current/PACKAGES.TXT" \ + "$(mirror_path 'file:///mnt/nas/' 'slackware64-current/PACKAGES.TXT')" +check "mirror_path no trailing slash" \ + "file:///mnt/nas/x/y" "$(mirror_path 'file:///mnt/nas' '/x/y')" + +# --- is_local_mirror --- +is_local_mirror "file:///mnt/nas"; check_rc "is_local file://" 0 $? +is_local_mirror "/mnt/nas"; check_rc "is_local bare abs" 0 $? +is_local_mirror "https://x/y"; check_rc "is_local https" 1 $? + +# --- strip_scheme --- +check "strip_scheme file://" "/mnt/nas" "$(strip_scheme 'file:///mnt/nas')" +check "strip_scheme bare" "/mnt/nas" "$(strip_scheme '/mnt/nas')" + +# --- fetch: local file resolves --- +echo hello > "$tmp/src.txt" +fetch "file://$tmp/src.txt" "$tmp/dst.txt"; check_rc "fetch local ok" 0 $? +check "fetch local content" "hello" "$(cat "$tmp/dst.txt" 2>/dev/null)" + +# --- fetch: missing local file errors --- +fetch "file://$tmp/nope.txt" "$tmp/x.txt"; check_rc "fetch local missing" 1 $? + +# --- txz_hash: stable + content-sensitive --- +mkdir -p "$tmp/pkgs" +printf a > "$tmp/pkgs/sbopkg-1.txz" +printf b > "$tmp/pkgs/tools-1.txz" +h1="$(txz_hash "$tmp/pkgs")" +h2="$(txz_hash "$tmp/pkgs")" +check "txz_hash stable" "$h1" "$h2" +printf c > "$tmp/pkgs/tools-1.txz" # change content +h3="$(txz_hash "$tmp/pkgs")" +check_rc "txz_hash changes on content" 0 "$([[ "$h1" != "$h3" ]] && echo 0 || echo 1)" +check "txz_hash empty dir" "" "$(txz_hash "$tmp/empty-nope")" + +# --- require_mount: missing mount errors (subshell to catch _err exit) --- +MIRROR="file://$tmp/no-such-root" +( require_mount current ) 2>/dev/null; check_rc "require_mount missing" 1 $? +mkdir -p "$tmp/mnt/slackware64-current" +MIRROR="file://$tmp/mnt" +( require_mount current ) 2>/dev/null; check_rc "require_mount present" 0 $? +MIRROR="https://example/x" +( require_mount current ) 2>/dev/null; check_rc "require_mount http skips" 0 $? + +echo "----" +echo "PASS: $pass FAIL: $fail" +[[ "$fail" -eq 0 ]] +``` + +- [ ] **Step 2: Run — expect PASS** + +Run: `bash .extras/image-builder/test-image-builder.sh` +Expected: final line `PASS: N FAIL: 0`, exit 0. + +If any FAIL, fix `lib.sh` (the test encodes the intended behavior). + +- [ ] **Step 3: Commit** + +```bash +git add .extras/image-builder/test-image-builder.sh +git commit -m 'image-builder: lib.sh self-check' +``` + +--- + +## Task 3: bootstrap.sh (base image) + +**Files:** +- Create: `.extras/image-builder/bootstrap.sh` +- Reference: forge `docker-images/scripts/bootstrap.sh` + +This is the forge `bootstrap.sh` adapted: source `config`+`lib.sh`, drop i586, +`file://` fetch via `fetch()`, LAN registry, `sbo-base` name, mount guard. + +- [ ] **Step 1: Write bootstrap.sh** + +Create `.extras/image-builder/bootstrap.sh`. Start from the forge original and apply exactly these changes; everything else (the `download_pkgtxt`, `find_package`, rootfs configuration, cleanup, Dockerfile-from-scratch blocks) is copied verbatim from the forge script: + +1. Header block: + +```sh +#!/bin/bash +# bootstrap.sh — build the sbo-base:{ver} image FROM scratch from NAS trees. +# Run as root (installpkg). Adapted from forge slackware/docker-images. +set -euo pipefail +HERE="$(cd "$(dirname "$0")" && pwd)" +source "${HERE}/config" +LOG_TAG=bootstrap +source "${HERE}/lib.sh" + +REGISTRY_IMAGE="${REGISTRY}/sbo-base" +[[ "${EUID}" -eq 0 ]] || _err "run as root." +mkdir -p "${HASH_DIR}" +``` + +2. Argument parsing: keep `--force` and `--version` only; **remove `--arch`** + (x86_64 fixed). Variant list comes from config: + +```sh +FORCE=false +OPT_VERSION="" +while [[ $# -gt 0 ]]; do + case "$1" in + --version) OPT_VERSION="$2"; shift 2 ;; + --force) FORCE=true; shift ;; + *) _err "unknown argument: $1" ;; + esac +done +if [[ -n "${OPT_VERSION}" ]]; then + BUILD_VARIANTS=("${OPT_VERSION}") +else + BUILD_VARIANTS=("${VARIANTS[@]}") +fi +``` + +3. Copy the forge `PACKAGES=( ... )` array verbatim (lines 80–187 of the forge + original — the full base package list). + +4. Replace `changelog_changed`: read ChangeLog via `fetch` from the NAS, + x86_64-only repo key: + +```sh +# changelog_changed VERSION -> 0 if changed (or --force), else 1 +changelog_changed() { + local version="$1" + local repo_key="slackware64-${version}" + local hash_file="${HASH_DIR}/${repo_key}.sha256" + local url; url="$(mirror_path "${MIRROR}" "${repo_key}/ChangeLog.txt")" + local tmp; tmp="$(mktemp)" + if ! fetch "${url}" "${tmp}"; then + rm -f "${tmp}" + _warn "${repo_key}: cannot read ChangeLog (${url}); skipping." + return 1 + fi + local live; live=$(sha256sum "${tmp}" | cut -d' ' -f1); rm -f "${tmp}" + if [[ "${FORCE}" == "true" ]]; then + echo "${live}" > "${hash_file}"; return 0 + fi + local stored=""; [[ -f "${hash_file}" ]] && stored=$(cat "${hash_file}") + if [[ "${live}" == "${stored}" ]]; then + _log "${repo_key}: ChangeLog unchanged; skipping."; return 1 + fi + _log "${repo_key}: ChangeLog changed."; echo "${live}" > "${hash_file}"; return 0 +} +``` + +5. Copy `download_pkgtxt` and `find_package` verbatim from the forge original, + but change the two `curl -sfL ... "${URL}" -o "${DEST}"` download lines in + `download_pkgtxt` to `fetch "${URL}" "${DEST}"` (drops the http assumption). + +6. `build_variant`: take `VERSION` only (no ARCH). Set: + +```sh +build_variant() { + local VERSION="$1" + require_mount "${VERSION}" + local REPO_KEY="slackware64-${VERSION}" + local PKG_PATH; PKG_PATH="$(mirror_path "${MIRROR}" "${REPO_KEY}")" + local TAG="${REGISTRY_IMAGE}:${VERSION}" + local SLACKPKG_MIRROR="https://slackware.nl/slackware/slackware64-${VERSION}/" + ... +} +``` + + The `SLACKPKG_MIRROR` written into the image stays an https slackware.nl URL + (it is used later by slackpkg inside the container, which has internet; the + NAS `file://` mount only feeds the host-side build). Keep the rest of the + forge `build_variant` body verbatim, except: + - the package **download loop** uses `fetch "${URL}" "${PKGCACHE}/${FILENAME}"` + instead of `curl`. + - the final `docker build ... && docker push` targets `${TAG}` as above. + +7. Main loop over `BUILD_VARIANTS`, one arg: + +```sh +for VERSION in "${BUILD_VARIANTS[@]}"; do + if changelog_changed "${VERSION}"; then + build_variant "${VERSION}" + fi +done +``` + +- [ ] **Step 2: Syntax check** + +Run: `bash -n .extras/image-builder/bootstrap.sh` +Expected: no output, exit 0. + +- [ ] **Step 3: shellcheck (if available)** + +Run: `shellcheck -x .extras/image-builder/bootstrap.sh || true` +Expected: no errors (warnings about unreachable `_err` exit are fine). If shellcheck absent, skip. + +- [ ] **Step 4: Commit** + +```bash +git add .extras/image-builder/bootstrap.sh +git commit -m 'image-builder: bootstrap.sh base image from NAS' +``` + +--- + +## Task 4: build-full-image.sh + +**Files:** +- Create: `.extras/image-builder/build-full-image.sh` +- Reference: forge `docker-images/scripts/build-full-image.sh` + +Forge `build-full-image.sh` adapted: source config+lib, drop i586, LAN +registry, `sbo-base`→`sbo-full` names. + +- [ ] **Step 1: Write build-full-image.sh** + +Create `.extras/image-builder/build-full-image.sh`. From the forge original, apply exactly: + +1. Header: + +```sh +#!/bin/bash +# build-full-image.sh — sbo-full:{ver} FROM sbo-base:{ver}, all series. +# Adapted from forge slackware/docker-images. No root needed. +set -euo pipefail +HERE="$(cd "$(dirname "$0")" && pwd)" +source "${HERE}/config" +LOG_TAG=full +source "${HERE}/lib.sh" + +BASE_IMAGE="${REGISTRY}/sbo-base" +FULL_IMAGE="${REGISTRY}/sbo-full" +DIGEST_LABEL="sbo.full.base-digest" +``` + +2. Arg parsing: `--force` and `--version` only (no `--arch`), variant list from + config — identical shape to bootstrap Task 3 Step 1.2, but `BUILD_VARIANTS`. + +3. `derive_tags VERSION`: + +```sh +derive_tags() { + local VERSION="$1" + BASE_TAG="${BASE_IMAGE}:${VERSION}" + FULL_TAG="${FULL_IMAGE}:${VERSION}" +} +``` + +4. Keep `needs_rebuild` verbatim from the forge original (it already takes + BASE_TAG/FULL_TAG args and uses `_log`/`_warn`/`FORCE`). + +5. `build_variant VERSION`: drop the ARCH/ARCH_SUFFIX lines; call + `derive_tags "${VERSION}"`. Keep the forge Dockerfile heredoc **verbatim** + (the `slackpkg update` + all-series install + cleanup + labels). Only the + `LABEL org.opencontainers.image.title/description` strings lose the + `(${ARCH})` suffix; leave them or hardcode `(x86_64)`. + +6. Main loop over `BUILD_VARIANTS`: + +```sh +for VERSION in "${BUILD_VARIANTS[@]}"; do + build_variant "${VERSION}" +done +``` + +- [ ] **Step 2: Syntax check** + +Run: `bash -n .extras/image-builder/build-full-image.sh` +Expected: exit 0. + +- [ ] **Step 3: shellcheck (if available)** + +Run: `shellcheck -x .extras/image-builder/build-full-image.sh || true` + +- [ ] **Step 4: Commit** + +```bash +git add .extras/image-builder/build-full-image.sh +git commit -m 'image-builder: build-full-image.sh' +``` + +--- + +## Task 5: build-sbo-testbuild.sh + +**Files:** +- Create: `.extras/image-builder/build-sbo-testbuild.sh` + +New script (no forge equivalent). FROM sbo-full, installpkg the two .txz, +gate on full digest AND .txz-set hash. + +- [ ] **Step 1: Write build-sbo-testbuild.sh** + +Create `.extras/image-builder/build-sbo-testbuild.sh`: + +```sh +#!/bin/bash +# build-sbo-testbuild.sh — sbo-testbuild:{ver} FROM sbo-full:{ver}. +# Adds sbopkg + sbo-maintainer-tools from prebuilt .txz in PKGDIR. +# Rebuilds when the full image OR the .txz set changes. +set -euo pipefail +HERE="$(cd "$(dirname "$0")" && pwd)" +source "${HERE}/config" +LOG_TAG=testbuild +source "${HERE}/lib.sh" + +FULL_IMAGE="${REGISTRY}/sbo-full" +TB_IMAGE="${REGISTRY}/sbo-testbuild" +DIGEST_LABEL="sbo.testbuild.full-digest" +PKGS_LABEL="sbo.testbuild.pkgs-hash" + +FORCE=false +OPT_VERSION="" +while [[ $# -gt 0 ]]; do + case "$1" in + --version) OPT_VERSION="$2"; shift 2 ;; + --force) FORCE=true; shift ;; + *) _err "unknown argument: $1" ;; + esac +done +if [[ -n "${OPT_VERSION}" ]]; then + BUILD_VARIANTS=("${OPT_VERSION}") +else + BUILD_VARIANTS=("${VARIANTS[@]}") +fi + +# Verify the required .txz are present before doing anything. +require_pkgs() { + [[ -d "${PKGDIR}" ]] || _err "PKGDIR missing: ${PKGDIR}" + compgen -G "${PKGDIR}/sbopkg-*.txz" >/dev/null \ + || _err "no sbopkg-*.txz in ${PKGDIR}" + compgen -G "${PKGDIR}/sbo-maintainer-tools-*.txz" >/dev/null \ + || _err "no sbo-maintainer-tools-*.txz in ${PKGDIR}" +} + +# needs_rebuild FULL_TAG TB_TAG PKGS_HASH -> 0 if rebuild needed +needs_rebuild() { + local full_tag="$1" tb_tag="$2" pkgs_hash="$3" + [[ "${FORCE}" == "true" ]] && { _log " --force."; return 0; } + + local full_digest + full_digest=$(docker inspect --format '{{index .RepoDigests 0}}' \ + "${full_tag}" 2>/dev/null || echo "") + [[ -n "${full_digest}" ]] || { _warn " no full digest; rebuilding."; return 0; } + + local rec_digest rec_pkgs + rec_digest=$(docker inspect \ + --format "{{index .Config.Labels \"${DIGEST_LABEL}\"}}" \ + "${tb_tag}" 2>/dev/null || echo "") + rec_pkgs=$(docker inspect \ + --format "{{index .Config.Labels \"${PKGS_LABEL}\"}}" \ + "${tb_tag}" 2>/dev/null || echo "") + + if [[ "${full_digest}" == "${rec_digest}" && "${pkgs_hash}" == "${rec_pkgs}" ]]; then + _log " full image and .txz set unchanged; skipping." + return 1 + fi + _log " full image or .txz set changed; rebuilding." + return 0 +} + +build_variant() { + local VERSION="$1" + local FULL_TAG="${FULL_IMAGE}:${VERSION}" + local TB_TAG="${TB_IMAGE}:${VERSION}" + _log "=== ${TB_TAG} ===" + + _log " Pulling ${FULL_TAG}..." + docker pull "${FULL_TAG}" + + local PKGS_HASH; PKGS_HASH="$(txz_hash "${PKGDIR}")" + + if ! needs_rebuild "${FULL_TAG}" "${TB_TAG}" "${PKGS_HASH}"; then + return 0 + fi + + local FULL_DIGEST + FULL_DIGEST=$(docker inspect --format '{{index .RepoDigests 0}}' "${FULL_TAG}") + + local WORKDIR; WORKDIR="$(mktemp -d /tmp/sbo-testbuild.XXXXXX)" + trap "rm -rf '${WORKDIR}'" RETURN + mkdir -p "${WORKDIR}/pkgs" + cp "${PKGDIR}"/sbopkg-*.txz "${PKGDIR}"/sbo-maintainer-tools-*.txz "${WORKDIR}/pkgs/" + + cat > "${WORKDIR}/Dockerfile" <<DOCKERFILE +FROM ${FULL_TAG} +LABEL maintainer="danix <danix@danix.xyz>" +COPY pkgs/*.txz /tmp/pkgs/ +RUN installpkg /tmp/pkgs/*.txz && rm -rf /tmp/pkgs +RUN /sbin/ldconfig +LABEL org.opencontainers.image.created="$(date -u +%Y-%m-%dT%H:%M:%SZ)" +LABEL org.opencontainers.image.title="SBo test-build env ${VERSION} (x86_64)" +LABEL org.opencontainers.image.description="Slackware ${VERSION} full + sbopkg + sbo-maintainer-tools" +LABEL slackware.version="${VERSION}" +LABEL ${DIGEST_LABEL}="${FULL_DIGEST}" +LABEL ${PKGS_LABEL}="${PKGS_HASH}" +CMD ["/bin/bash"] +DOCKERFILE + + local BUILD_FLAGS=() + [[ "${FORCE}" == "true" ]] && BUILD_FLAGS+=(--no-cache) + _log " Building ${TB_TAG}..." + docker build "${BUILD_FLAGS[@]}" -t "${TB_TAG}" "${WORKDIR}" + _log " Pushing ${TB_TAG}..." + docker push "${TB_TAG}" + _log "=== Done: ${TB_TAG} ===" +} + +require_pkgs +for VERSION in "${BUILD_VARIANTS[@]}"; do + build_variant "${VERSION}" +done +``` + +- [ ] **Step 2: Syntax check** + +Run: `bash -n .extras/image-builder/build-sbo-testbuild.sh` +Expected: exit 0. + +- [ ] **Step 3: shellcheck (if available)** + +Run: `shellcheck -x .extras/image-builder/build-sbo-testbuild.sh || true` + +- [ ] **Step 4: Commit** + +```bash +git add .extras/image-builder/build-sbo-testbuild.sh +git commit -m 'image-builder: build-sbo-testbuild.sh' +``` + +--- + +## Task 6: per-variant isolation + +The spec requires one variant failing not to block the other, non-zero exit at +the end. Add this to all three scripts' main loops. + +**Files:** +- Modify: `.extras/image-builder/bootstrap.sh` (main loop) +- Modify: `.extras/image-builder/build-full-image.sh` (main loop) +- Modify: `.extras/image-builder/build-sbo-testbuild.sh` (main loop) + +- [ ] **Step 1: Wrap each main loop** + +In each script, replace the plain `for VERSION ...` main loop with an +isolation wrapper. For **bootstrap.sh** (which also has the changelog gate): + +```sh +rc=0 +for VERSION in "${BUILD_VARIANTS[@]}"; do + if changelog_changed "${VERSION}"; then + if ! ( build_variant "${VERSION}" ); then + _warn "variant ${VERSION} failed; continuing." + rc=1 + fi + fi +done +exit "${rc}" +``` + +For **build-full-image.sh** and **build-sbo-testbuild.sh** (no changelog gate): + +```sh +rc=0 +for VERSION in "${BUILD_VARIANTS[@]}"; do + if ! ( build_variant "${VERSION}" ); then + _warn "variant ${VERSION} failed; continuing." + rc=1 + fi +done +exit "${rc}" +``` + +Note: `build_variant` runs in a subshell `( )` so a `set -e` abort inside it +(or an `_err` exit) is contained and the loop proceeds to the next variant. +For build-sbo-testbuild.sh keep the `require_pkgs` call **before** the loop +(a missing .txz is fatal for all variants, not per-variant). + +- [ ] **Step 2: Syntax check all three** + +Run: `for f in bootstrap build-full-image build-sbo-testbuild; do bash -n .extras/image-builder/$f.sh; done` +Expected: exit 0, no output. + +- [ ] **Step 3: Re-run lib self-check (unaffected, sanity)** + +Run: `bash .extras/image-builder/test-image-builder.sh` +Expected: `FAIL: 0`. + +- [ ] **Step 4: Commit** + +```bash +git add .extras/image-builder/bootstrap.sh .extras/image-builder/build-full-image.sh .extras/image-builder/build-sbo-testbuild.sh +git commit -m 'image-builder: per-variant isolation, non-zero exit on failure' +``` + +--- + +## Task 7: README + +**Files:** +- Create: `.extras/image-builder/README` + +- [ ] **Step 1: Write README** + +Create `.extras/image-builder/README` with the VM-side setup, verbatim: + +``` +sbo-testbuild image builder +=========================== + +Builds the docker images that .extras/test-build consumes: + docker.noland.dnx:5000/sbo-testbuild:current + docker.noland.dnx:5000/sbo-testbuild:15.0 + +Three scripts, chained (see .extras/docs/specs/2026-07-13-image-builder-design.md): + bootstrap.sh sbo-base:{ver} FROM scratch, base pkgs from NAS + build-full-image.sh sbo-full:{ver} FROM base, all series + build-sbo-testbuild.sh sbo-testbuild:{ver} FROM full, + sbopkg + tools + +All settings live in ./config. + +VM setup (docker.noland.dnx, Slackware x86_64, 4 vCPU / 4 GB / 80 GB) +-------------------------------------------------------------------- +1. Install docker; enable the daemon. + +2. NFS-mount the two NAS trees read-only, named to match: + /mnt/nas/slackware64-current -> -current mirror tree + /mnt/nas/slackware64-15.0 -> 15.0 mirror tree + Each is a full mirror (PACKAGES.TXT, ChangeLog.txt, slackware64/, patches/, + extra/). Root must be able to read them (bootstrap runs installpkg as root). + +3. Run a LAN registry: + docker run -d --restart=always -p 5000:5000 \ + -v /opt/registry/data:/var/lib/registry --name registry registry:2 + +4. Mark the registry insecure (plain HTTP) on the VM AND every pulling client + (this dev box, the buildsystem VM). In /etc/docker/daemon.json: + { "insecure-registries": ["docker.noland.dnx:5000"] } + then restart docker. + +5. Drop the two prebuilt packages (built once, re-drop on upstream bumps): + /opt/sbo-testbuild/pkgs/sbopkg-*.txz + /opt/sbo-testbuild/pkgs/sbo-maintainer-tools-*.txz + +6. Install the nightly cron (root): + 0 4 * * * /path/to/.extras/image-builder/bootstrap.sh >> /var/log/sbo-testbuild.log 2>&1 + 15 4 * * * /path/to/.extras/image-builder/build-full-image.sh >> /var/log/sbo-testbuild.log 2>&1 + 30 4 * * * /path/to/.extras/image-builder/build-sbo-testbuild.sh >> /var/log/sbo-testbuild.log 2>&1 + +7. Ensure docker.noland.dnx resolves on the LAN (static IP or DNS). + +Manual first run +---------------- + ./bootstrap.sh --version current --force + ./build-full-image.sh --version current --force + ./build-sbo-testbuild.sh --version current --force +Then confirm: + docker pull docker.noland.dnx:5000/sbo-testbuild:current + docker run --rm docker.noland.dnx:5000/sbo-testbuild:current sbopkg -V + +Flags: --force (rebuild unconditionally), --version <current|15.0> (one variant). + +Tests +----- + bash test-image-builder.sh # pure-logic self-check, no docker +``` + +- [ ] **Step 2: Commit** + +```bash +git add .extras/image-builder/README +git commit -m 'image-builder: README with VM setup checklist' +``` + +--- + +## Task 8: executable bits + final check + +**Files:** +- Modify (mode): the four `.sh` scripts + +- [ ] **Step 1: chmod +x the scripts** + +```bash +chmod +x .extras/image-builder/bootstrap.sh \ + .extras/image-builder/build-full-image.sh \ + .extras/image-builder/build-sbo-testbuild.sh \ + .extras/image-builder/test-image-builder.sh +``` + +- [ ] **Step 2: Run the self-check once more** + +Run: `bash .extras/image-builder/test-image-builder.sh` +Expected: `PASS: N FAIL: 0`, exit 0. + +- [ ] **Step 3: Syntax-check all four** + +Run: `for f in .extras/image-builder/*.sh; do bash -n "$f" || echo "SYNTAX FAIL: $f"; done` +Expected: no `SYNTAX FAIL` lines. + +- [ ] **Step 4: Commit** + +```bash +git add .extras/image-builder +git commit -m 'image-builder: mark scripts executable' +``` + +--- + +## Deferred (not in this plan) + +- **Real end-to-end chain** on the VM (bootstrap→full→testbuild producing a + pushed `sbo-testbuild:current`): the user runs it once the VM/registry/mounts + exist, and reports back. Mirrors `.extras/test-build` Task 10. +- **`installed_in_base`** in `.extras/test-build`: now unblockable (query the + built image's package db) but tracked separately in the project memory + `test-build-image-builder-todo`. +- **i586** variants. +- **Registry auth/TLS** (LAN plain HTTP for now). +``` |
