# sb-builder bundled deps (phase 4b, part 2), implementation plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** `hint-update` handles packages with several downloads: every URL the bump changed gets a new md5, and a package in the helper's `[manifests]` (neovim) has its extra `DOWNLOAD` URLs moved to the versions of upstream's deps list. **Architecture:** mkhint's url mode ported into `remote/sbb-remote` as pure text functions (`_url_repo`, `_url_stem`, `_url_version`, `_match`, `reconcile`) called from `apply_version`, which gains a `manifest` argument fed from config `[manifests]`. No GUI change. **Tech Stack:** Python 3.11+ stdlib (`re`, `urllib`, `hashlib`, `tomllib`). Spec: `docs/superpowers/specs/2026-10-20-sb-builder-bundled-deps-design.md`. Every code step below was run before this plan was written: the diffs apply to `master` at `891a918` with `git apply` and the suite ends at 128 tests, OK. Two `ResourceWarning: subprocess ... is still running` lines are expected (detached `_run`). `reconcile` was also run against the live neovim deps lists with the 14 real URLs of the neovim `.info`: v0.12.4 reports all current ("manifest has 7 deps not in the hint"), v0.11.0 moves 10 deps (luajit, luv, the tree-sitters, utf8proc). ## Files | File | Change | |---|---| | `remote/sbb-remote` | `[manifests]` config, `manifest_packages` and its refusal gone, url functions, `reconcile`, `apply_version` md5 for changed URLs and manifest, VERSION 0.8.0 | | `remote/config.example.toml` | `[manifests]` in place of `manifest_packages` | | `sb_builder/__init__.py` | VERSION 0.8.0 | | `tests/test_remote.py` | config, url matching on neovim's real shapes, reconcile, apply with manifest, feroxbuster shape | | `README.md`, `AGENTS.md` | neovim no longer refused, mkhint no longer used | --- ### Task 1: tests **Files:** - Test: `tests/test_remote.py` - [ ] **Step 1: Apply the test diff** ```diff diff --git i/tests/test_remote.py w/tests/test_remote.py index 57d492c..30bdc73 100644 --- i/tests/test_remote.py +++ w/tests/test_remote.py @@ -135,6 +135,12 @@ class ConfigTest(Base): with self.assertRaisesRegex(sbb.Error, "bogus: unknown key"): sbb.load_config(self.write("bogus = 1\n")) + def test_manifests(self): + self.assertEqual(sbb.load_config(self.write("[manifests]\nx = \"u{VERSION}\"\n"))["manifests"], + {"x": "u{VERSION}"}) + with self.assertRaisesRegex(sbb.Error, "manifests: expected strings"): + sbb.load_config(self.write("[manifests]\nx = 1\n")) + def test_wrong_type(self): with self.assertRaisesRegex(sbb.Error, "state_dir: expected str"): sbb.load_config(self.write("state_dir = 1\n")) @@ -302,6 +308,73 @@ class ApplyVersionTest(Base): self.apply(f'VERSION="1.0"\nDOWNLOAD="{self.www.url}/x-1.0.tgz"\nMD5SUM="m"\n', "3.0") +GH = "https://github.com" +# neovim 0.12.4: hint URLs (SBo shapes) and their deps.txt counterparts +NVIM_HINT = [f"{GH}/neovim/neovim/archive/v0.12.4/neovim-0.12.4.tar.gz", + f"{GH}/luajit/LuaJIT/archive/fbb36bb/LuaJIT-fbb36bb6bfa88716a47c58bcf9ce9f2ef752abac.tar.gz", + f"{GH}/luvit/luv/releases/download/1.52.1-0/luv-1.52.1-0.tar.gz", + f"{GH}/lunarmodules/lua-compat-5.3/archive/v0.13/lua-compat-5.3-0.13.tar.gz", + f"{GH}/tree-sitter/tree-sitter/archive/v0.26.7/tree-sitter-0.26.7.tar.gz", + "https://www.inf.puc-rio.br/~roberto/lpeg/lpeg-1.1.0.tar.gz", + f"{GH}/JuliaStrings/utf8proc/archive/v2.11.3/utf8proc-2.11.3.tar.gz"] +NVIM_DEPS = [f"{GH}/luajit/luajit/archive/fbb36bb6bfa88716a47c58bcf9ce9f2ef752abac.tar.gz", + f"{GH}/luvit/luv/archive/1.52.1-0.tar.gz", + f"{GH}/lunarmodules/lua-compat-5.3/archive/v0.13.tar.gz", + f"{GH}/tree-sitter/tree-sitter-c/archive/v0.24.1.tar.gz", + f"{GH}/tree-sitter/tree-sitter/archive/v0.26.7.tar.gz", + f"{GH}/neovim/deps/raw/d495ee6f79e7962a53ad79670cb92488abe0b9b4/opt/lpeg-1.1.0.tar.gz", + f"{GH}/juliastrings/utf8proc/archive/v2.11.3.tar.gz"] + + +class BundledDepsTest(Base): + def test_hint_and_manifest_shapes_match_with_equal_versions(self): + for url in NVIM_HINT[1:]: + m = sbb._match(url, NVIM_DEPS) + with self.subTest(url=url): + self.assertIsNotNone(m) + self.assertEqual(sbb._url_version(url), sbb._url_version(m)) + self.assertIsNone(sbb._match(NVIM_HINT[0], NVIM_DEPS)) + self.assertEqual([sbb._url_version(u) for u in NVIM_HINT[2:4]], ["1.52.1_0", "0.13"]) + self.assertEqual(sbb._dep_name(NVIM_HINT[5]), "lpeg") + + def test_reconcile(self): + hint = 'DOWNLOAD="' + " \\\n ".join(NVIM_HINT[:3] + [f"{GH}/a/gone/archive/v1/gone-1.tar.gz"]) + '"\n' + deps = "\n".join(f"X_URL {u}\nX_SHA256 ab" for u in NVIM_DEPS).replace("1.52.1-0", "1.53.0-0") + log = [] + out = sbb.reconcile(hint, deps, log.append) + self.assertEqual(sbb.hint_tokens(out, "DOWNLOAD"), + NVIM_HINT[:2] + [f"{GH}/luvit/luv/archive/1.53.0-0.tar.gz", + f"{GH}/a/gone/archive/v1/gone-1.tar.gz"]) + self.assertEqual(log, ["luv 1.52.1_0 -> 1.53.0_0", "gone no manifest match, left as is", + "manifest has 5 deps not in the hint"]) + self.assertIs(sbb.reconcile(hint, deps.replace("1.53.0-0", "1.52.1-0"), log.append), hint) + with self.assertRaisesRegex(sbb.Error, "no URLs"): + sbb.reconcile(hint, "moved\n", log.append) + + def test_apply_version_follows_manifest(self): + www = Served(self, {"x-2.0.tgz": b"two", "dep-0.6.tgz": b"dep11", "keep-3.tgz": b"keep"}) + u, log = www.url, [] + with open(os.path.join(self.tmp, "www", "deps-2.0.txt"), "w") as f: + f.write(f"DEP_URL {u}/dep-0.6.tgz\nKEEP_URL {u}/keep-3.tgz\n") + text = (f'VERSION="1.0"\nDOWNLOAD="{u}/x-1.0.tgz \\\n {u}/dep-0.5.tgz \\\n {u}/keep-3.tgz"\n' + 'MD5SUM="a \\\n b \\\n c"\n') + out = sbb.apply_version(text, "", "2.0", False, log.append, u + "/deps-{VERSION}.txt") + self.assertEqual(out, f'VERSION="2.0"\nDOWNLOAD="{u}/x-2.0.tgz \\\n {u}/dep-0.6.tgz \\\n' + f' {u}/keep-3.tgz"\nMD5SUM="{md5(b"two")} \\\n {md5(b"dep11")} \\\n c"\n') + self.assertEqual(log, [f"manifest {u}/deps-2.0.txt", "dep 0.5 -> 0.6", + f"downloading {u}/x-2.0.tgz", f"downloading {u}/dep-0.6.tgz"]) + with self.assertRaises(OSError): # no deps list at 3.0: the package fails, hint untouched + sbb.apply_version(text, "", "3.0", False, log.append, u + "/deps-{VERSION}.txt") + + def test_changed_continuation_url_gets_its_md5(self): + www = Served(self, {"f-2.0.tgz": b"two", "f-2.0-vendor.tgz": b"vendor"}) + u, log = www.url, [] + text = f'VERSION="1.0"\nDOWNLOAD="{u}/f-1.0.tgz \\\n {u}/f-1.0-vendor.tgz"\nMD5SUM="a \\\n b"\n' + self.assertIn(f'MD5SUM="{md5(b"two")} \\\n {md5(b"vendor")}"', + sbb.apply_version(text, "", "2.0", False, log.append)) + self.assertEqual(log, [f"downloading {u}/f-2.0.tgz", f"downloading {u}/f-2.0-vendor.tgz"]) + + class HintJobsTest(Base): def setUp(self): super().setUp() @@ -349,14 +422,12 @@ class HintJobsTest(Base): ("hint-update", ["nope=1"], "no such package"), ("hint-update", ["x=1", "x=2"], "twice"), ("hint-update", ["x=1", "--force"], "unknown option"), - ("hint-update", ["neovim=1"], "bundled deps"), ("hint-update", ["shadowed=1"], "not managed"), ("hint-new", [], "hint-new "), ("hint-new", ["x", "2.0"], "no such package: 2.0"), # no version any more ("hint-new", ["x", "x"], "twice")): with self.subTest(kind=kind, args=args), self.assertRaisesRegex(sbb.Error, why): sbb.KINDS[kind](self.cfg, args) - sbb.KINDS["hint-new"](self.cfg, ["neovim"]) # no version: no bundled deps to touch self.make_pkg("misc/y", self.info, hint="") with self.assertRaisesRegex(sbb.Error, "y: hint exists"): sbb.KINDS["hint-new"](self.cfg, ["x", "y"]) # one with a hint refuses the whole job ``` - [ ] **Step 2: Run, expect failures** Run: `python3 -m unittest discover -s tests 2>&1 | tail -3` Expected: FAILED, `AttributeError: module 'sbb' has no attribute '_match'` and friends, `manifests: unknown key`. ### Task 2: helper and version **Files:** - Modify: `remote/sbb-remote`, `remote/config.example.toml`, `sb_builder/__init__.py` - [ ] **Step 1: Apply the helper diff** ```diff diff --git i/remote/config.example.toml w/remote/config.example.toml index 0fbe801..18ade4c 100644 --- i/remote/config.example.toml +++ w/remote/config.example.toml @@ -12,5 +12,9 @@ hintdir = "/etc/slackrepo/SBo-danix/hintfiles" default_hintdir = "" # searched after hintdir when set genreprc = "/root/.genreprc" # REPOSOWNERGPG names the signing key slackrepo_logdir = "/var/log/slackrepo/SBo-danix" -manifest_packages = ["neovim"] # bundled deps: hint-update refuses them (phase 4b) phantom_deps_file = "/etc/sb-builder/phantom-deps" # deps -current ships; one per line, # comments + +# bundled deps: package = its upstream deps list URL, {VERSION} = the new version. +# hint-update moves the package's extra DOWNLOAD URLs to the versions listed there. +[manifests] +neovim = "https://raw.githubusercontent.com/neovim/neovim/v{VERSION}/cmake.deps/deps.txt" diff --git i/remote/sbb-remote w/remote/sbb-remote index c37b44c..07c2a55 100755 --- i/remote/sbb-remote +++ w/remote/sbb-remote @@ -40,7 +40,7 @@ import time import tomllib import urllib.request -VERSION = "0.7.0" +VERSION = "0.8.0" DEFAULTS = { "state_dir": "/var/lib/sb-builder", "repo_dir": "/var/lib/sbopkg/SBo-danix", @@ -53,7 +53,8 @@ DEFAULTS = { "default_hintdir": "", "genreprc": "/root/.genreprc", "slackrepo_logdir": "/var/log/slackrepo/SBo-danix", - "manifest_packages": ["neovim"], # bundled deps: left to mkhint until phase 4b + # bundled deps: package -> deps list URL, {VERSION} the new version + "manifests": {"neovim": "https://raw.githubusercontent.com/neovim/neovim/v{VERSION}/cmake.deps/deps.txt"}, "phantom_deps_file": "/etc/sb-builder/phantom-deps", # deps -current ships: DELREQUIREd } FLAGS = ("--yes", "--no") @@ -90,9 +91,10 @@ def load_config(path=None): if type(v) is not type(DEFAULTS[k]): raise Error(f"config: {k}: expected {type(DEFAULTS[k]).__name__}") cfg[k] = v - for k in ("overlays", "manifest_packages"): - if not all(isinstance(o, str) for o in cfg[k]): - raise Error(f"config: {k}: expected list of strings") + if not all(isinstance(o, str) for o in cfg["overlays"]): + raise Error("config: overlays: expected list of strings") + if not all(isinstance(u, str) for u in cfg["manifests"].values()): + raise Error("config: manifests: expected strings") return cfg @@ -187,8 +189,6 @@ def hint_args(cfg, kind, args): if pkg in seen: raise Error(f"{pkg} named twice") seen.add(pkg) - if ver is not None and pkg in cfg["manifest_packages"]: - raise Error(f"{pkg}: bundled deps: use mkhint until phase 4b") if kind == "hint-new" and os.path.exists(target): raise Error(f"{pkg}: hint exists") return pairs, no_dl @@ -710,28 +710,128 @@ def _md5_url(url): return h.hexdigest() -def apply_version(text, info, new, no_dl, log): +def _fetch(url): + req = urllib.request.Request(url, headers={"User-Agent": "sbb-remote"}) + with urllib.request.urlopen(req, timeout=DL_TIMEOUT) as r: + return r.read().decode(errors="replace") + + +# ── bundled deps: a hint's extra DOWNLOAD URLs against upstream's deps list ── +# mkhint's url mode. Both sides of a pair reduce to "the part that isn't the +# name", so one version in two URL shapes (v0.13.tar.gz, lua-compat-5.3-0.13.tar.gz) +# compares equal. + +MANIFEST_URL = re.compile(r"""https?://[^\s"'<>()]+""") +ARCHIVE = re.compile(r"\.(?:tar\.(?:gz|xz|bz2|zst)|tgz|zip)$") + + +def _url_repo(url): + """owner/repo of a GitHub URL, lowercased (GitHub ignores case); "" elsewhere. + + neovim/deps is a shared blob repo that names the host, not the dep: "". + """ + m = re.search(r"github\.com/([^/]+)/([^/]+)", url) + repo = f"{m.group(1)}/{m.group(2).removesuffix('.git')}".lower() if m else "" + return "" if repo.endswith("/deps") else repo + + +def _url_base(url): + return ARCHIVE.sub("", url.rsplit("/", 1)[-1]) + + +def _url_stem(url): + """The basename without archive extension and trailing - or -.""" + m = re.match(r"(.+)-(?:[vV]?\d.*|[0-9a-f]{7,})$", _url_base(url)) + return m.group(1) if m else _url_base(url) + + +def _url_version(url): + """On GitHub the basename past "-" (the whole basename when it lacks + it); elsewhere its trailing - or -. No leading v, '-' as '_'. + + ponytail: off GitHub the name is a guess; a "-" + basename there misparses. Add a name hint if such a dep ever appears. + """ + base, repo = _url_base(url), _url_repo(url).split("/")[-1] + if repo: + ver = base[len(repo) + 1:] if base.lower().startswith((repo + "-", repo + "_")) else base + else: + m = re.search(r"-([vV]?\d[^/]*)$", base) or re.search(r"-([0-9a-f]{7,})$", base) + ver = m.group(1) if m else base + return ver.removeprefix("v").removeprefix("V").replace("-", "_") + + +def _dep_name(url): + return _url_repo(url).split("/")[-1] or _url_stem(url) + + +def _match(url, found): + """The manifest URL for a hint URL: same GitHub repo, else same stem.""" + repo = _url_repo(url) + m = repo and next((f for f in found if _url_repo(f) == repo), None) + return m or next((f for f in found if _url_stem(f) == _url_stem(url)), None) + + +def reconcile(text, manifest, log): + """DOWNLOAD's extra URLs moved to the versions of manifest, a deps list's text. + + The first URL is the package itself: the version swap moved it already. + """ + found = sorted(set(MANIFEST_URL.findall(manifest))) + if not found: + raise Error("manifest has no URLs") + urls = hint_tokens(text, "DOWNLOAD") + used = {_match(urls[0], found)} if urls else set() + for i, url in enumerate(urls[1:], 1): + m = _match(url, found) + if m is None: + log(f"{_dep_name(url)} no manifest match, left as is") + continue + used.add(m) + old, new = _url_version(url), _url_version(m) + if old != new: + log(f"{_dep_name(url)} {old} -> {new}") + urls[i] = m + if extra := len(set(found) - used): + log(f"manifest has {extra} deps not in the hint") + if urls == hint_tokens(text, "DOWNLOAD"): + return text + return hint_set(text, "DOWNLOAD", " \\\n ".join(urls)) + + +def apply_version(text, info, new, no_dl, log, manifest=None): """The hint text at version new (None: keep the version), log(str) per download. - An unpinned hint (DELREQUIRES only) gets the .info pin first. Downloads the - first URL of each DOWNLOAD pair for its md5; continuation URLs keep theirs, - as in mkhint. no_dl: no download, DOWNLOAD="no" in place of every - DOWNLOAD*/MD5SUM* line, since DOWNLOAD_$ARCH would override it. + An unpinned hint (DELREQUIRES only) gets the .info pin first. manifest: the + package's deps list URL template; its extra DOWNLOAD URLs follow it. + Downloads the first URL of each DOWNLOAD pair and every URL the bump + changed, for its md5; the others keep theirs. no_dl: no download, + DOWNLOAD="no" in place of every DOWNLOAD*/MD5SUM* line, since + DOWNLOAD_$ARCH would override it. """ if not hint_tokens(text, "VERSION"): text = _pin(info) + ("" if hint_tokens(text, "ARCH") else 'ARCH="x86_64"\n') + text + before = {dl: hint_tokens(text, dl) for dl, _ in DL_PAIRS} if new is not None: text = swap_version(text, hint_tokens(text, "VERSION")[0], new) if no_dl: return hint_drop(text, r"(?:DOWNLOAD|MD5SUM)\w*") + 'DOWNLOAD="no"\n' if new is None: return text + if manifest: + url = manifest.replace("{VERSION}", new) + log(f"manifest {url}") + text = reconcile(text, _fetch(url), log) for dl, sums in DL_PAIRS: - urls = hint_tokens(text, dl) + urls, old = hint_tokens(text, dl), before[dl] if not urls or urls[0] in SKIP_URLS: continue - log(f"downloading {urls[0]}") - text = hint_set(text, sums, " \\\n ".join([_md5_url(urls[0]), *hint_tokens(text, sums)[1:]])) + md5s = (hint_tokens(text, sums) + [""] * len(urls))[:len(urls)] + for i, url in enumerate(urls): + if i == 0 or i >= len(old) or url != old[i]: + log(f"downloading {url}") + md5s[i] = _md5_url(url) + text = hint_set(text, sums, " \\\n ".join(md5s)) return text @@ -864,7 +964,8 @@ def cmd_hint(cfg, kind=None, *args): if text is None: text = new_hint(info) _save(target, apply_version(text, info, ver, no_dl, - lambda s, p=pkg: print(f"{p}: {s}", flush=True))) + lambda s, p=pkg: print(f"{p}: {s}", flush=True), + cfg["manifests"].get(pkg))) print(f"{pkg}: ok", flush=True) ok.append(pkg) except (Error, OSError) as e: diff --git i/sb_builder/__init__.py w/sb_builder/__init__.py index f4a1f98..e4a5697 100644 --- i/sb_builder/__init__.py +++ w/sb_builder/__init__.py @@ -16,4 +16,4 @@ """sb-builder: Qt6 GUI driving the SlackBuild tooling on a build VM.""" -VERSION = "0.7.0" # must match remote/sbb-remote VERSION +VERSION = "0.8.0" # must match remote/sbb-remote VERSION ``` - [ ] **Step 2: Run the suite** Run: `python3 -m unittest discover -s tests 2>&1 | tail -3` Expected: `Ran 128 tests`, `OK`. - [ ] **Step 3: Commit** ```bash git add remote/sbb-remote remote/config.example.toml sb_builder/__init__.py tests/test_remote.py git commit -m "sbb-remote 0.8.0: bundled deps from upstream's deps list" ``` ### Task 3: docs **Files:** - Modify: `README.md`, `AGENTS.md` - [ ] **Step 1: Apply the docs diff** ```diff diff --git i/AGENTS.md w/AGENTS.md index 9f92d71..7e28e0a 100644 --- i/AGENTS.md +++ w/AGENTS.md @@ -37,8 +37,10 @@ Design and phasing: `docs/superpowers/specs/`. Phase 1 (core) is jobs run the helper's own `_hint` command): text edits, `.bak` before each write, a final `SBB-RESULT {json}` log line read back with `job-result`. The phantom deps list is the helper's `phantom_deps_file`; sbshadow runs - `sbb-remote fix-current-now` after each sync. mkhint stays on the VM only - for neovim (bundled deps, `manifest_packages`), until phase 4b part 2. + `sbb-remote fix-current-now` after each sync. Bundled deps: a package in + the helper's `[manifests]` (neovim) has its extra `DOWNLOAD` URLs moved to + the versions of upstream's deps list on `hint-update`. mkhint is no longer + used; removing it from the VM is phase 4b part 3. - Jobs never get input. Any tool run as a job must be non-interactive (`--yes`/`--no` style flags). If a tool needs a flag it lacks, add it in that tool's own repo (e.g. `slackrepo_setup` for sbshadow) rather than diff --git i/README.md w/README.md index a686197..bb1ff56 100644 --- i/README.md +++ w/README.md @@ -78,8 +78,10 @@ reopens it. it is official): add names, or tick suggestions, the required names no package of the tree has; **Apply to hints** writes them into the `DELREQUIRES` of every package that requires them (sbshadow does the same - after each sync). neovim is refused for now: its bundled deps still need - mkhint on the VM. + after each sync). An update downloads every URL the bump changed for its + md5; a package with bundled deps (helper config `[manifests]`, neovim by + default) first has its extra downloads moved to the versions upstream's + deps list names. - **Results**: slackrepo's last build attempt per package, newest first, filterable by result. **Show build log** (or double-click) opens the end of the log of the package's last attempt (slackrepo keeps one per command: ``` - [ ] **Step 2: Commit and push** ```bash git add README.md AGENTS.md docs/superpowers/plans/2026-10-20-sb-builder-bundled-deps.md git commit -m "Document bundled deps; phase 4b part 2 plan" git push ``` ### Task 4: deploy (ask first) - [ ] **Step 1: Drop the obsolete key on the VM**, else 0.8.0 refuses its config (`manifest_packages: unknown key`): ```bash ssh buildsystem "sed -i '/^manifest_packages/d' /etc/sb-builder/config.toml" ``` - [ ] **Step 2: Deploy**: `./install.sh --remote`, then `ssh buildsystem sbb-remote version` reports `0.8.0`. - [ ] **Step 3: Manual check (user)**: Hints, tick neovim, update to the current upstream version; the job log lists `manifest ...`, any ` -> ` lines and a download per changed URL; the diff dialog shows the moved `DOWNLOAD`/`MD5SUM` lines.