# SHA-mode bundle reconcile Date: 2026-07-09 Status: Approved, ready for implementation plan ## Problem mkhint's bundle-reconcile (neovim) tracks bundled deps against a single flat upstream manifest (`cmake.deps/deps.txt`, `NAME_URL`/`NAME_SHA256` pairs) and rewrites a `DOWNLOAD` line when the dep's upstream *version* differs. openvino bundles its deps as **git submodules** pinned by **commit SHA**, not by version. There is no flat upstream manifest. Two of the pinned commits sit **off-tag** (ittapi, protobuf pin commits that are not any release tag), so resolving a SHA back to a version is unreliable and sometimes impossible. The existing version-based reconcile cannot handle this package. The user maintains their own copy of the openvino SlackBuild and will rewrite it (and the `.info`) so every bundled dep is referenced by a **full 40-char commit SHA** in a `github.com///archive//...` `DOWNLOAD` URL. This makes SHA-to-SHA comparison the whole check: no version derivation needed. ## Goal Add a second reconcile mode ("sha" mode) to mkhint that, for a configured package, checks each bundled dep's hint SHA against the SHA that the primary project currently pins for that submodule (via the GitHub contents API), and rewrites plus re-md5s any dep whose SHA moved. The primary package version still bumps through the normal `--check` path; this feature only reconciles the bundled deps, exactly as the neovim path does. This is **Job A: SHA drift** — are my pinned dep SHAs stale? A companion **Job B: set drift** (does openvino's submodule set still match my bundle?) is specified below. The two are kept separate: A rewrites lines, B is FYI-only. ## Non-goals - No SHA-to-version resolution. SHAs are compared as opaque identities. - No SlackBuild editing by mkhint. The user's rewritten SlackBuild must read dep SHAs from the hint's `DOWNLOAD` lines (no baked-in `COMMIT_*`/`VERSION_*` vars), so a hint rewrite actually drives the build. This is a prerequisite the user owns; mkhint's domain ends at the hint file. - No auto add/remove of bundled deps. Job B only *reports* set drift; adding or removing a bundled dep is a SlackBuild + `.info` edit the user owns. Note: Job A needs no `.gitmodules` (manifest names the paths). Job B *does* fetch `.gitmodules`, but only to diff the submodule set across versions, never to drive the SHA reconcile. ## Manifest format `BUNDLE_MANIFEST_FILE` moves to an explicit **3-field** shape, ` `, applied to ALL entries for consistency. Field 2 is the mode discriminator; the parser branches on it. Existing url-manifest (neovim) migrates to explicit `url` mode: ``` neovim url https://raw.githubusercontent.com/neovim/neovim/{VERSION}/cmake.deps/deps.txt ``` (`` for `url` mode is the `{VERSION}`-templated manifest URL, unchanged from today's field-2 content. Only the explicit `url` keyword is new.) New `sha` mode: ``` openvino sha github:openvinotoolkit/openvino {VERSION} \ onednn=src/plugins/intel_cpu/thirdparty/onednn \ onednn_gpu=src/plugins/intel_gpu/thirdparty/onednn_gpu \ mlas=src/plugins/intel_cpu/thirdparty/mlas \ flatbuffers=thirdparty/flatbuffers/flatbuffers \ onnx=thirdparty/onnx/onnx \ ittapi=thirdparty/ittapi/ittapi \ protobuf=thirdparty/protobuf/protobuf ``` `sha`-mode `` = `

...`: - ``: `owner/repo` of the primary project (accepts a `github:` prefix, stripped). Used to build the contents-API base URL. - ``: the ref at which submodule SHAs are queried. `{VERSION}` is substituted with the primary hint's current VERSION (e.g. `2024.4.1`). - `...`: one pair per tracked dep. `path` is the submodule path (the contents-API path). `name` is a **report label only** — it does not have to match the repo. The dep's repo (hence which `DOWNLOAD` line to rewrite) comes from the contents-API response field `submodule_git_url`, not from `name`. The manifest is plain whitespace-delimited text, hand-edited, a handful of lines. No JSON, no jq-to-read-own-config. (Considered and rejected: JSON buys nesting we don't need and adds jq as a hard dependency in the config hot path.) ## Data flow Runs from the same trigger as the neovim reconcile: automatically when the primary changed this run, otherwise only under `--force`. `--new` on a sha-mode package prints the report without changing the hint (mirrors existing `--new` reconcile behavior). For a sha-mode package: ``` version <- primary hint VERSION, substituted into version-template -> ref api_base <- https://api.github.com/repos//contents for each (name, path) in manifest deps: resp <- GET api_base/?ref= (authed if token) upstream_sha <- resp.sha (jq '.sha') dep_repo <- resp.submodule_git_url -> owner/repo (strip host + .git) line <- hint DOWNLOAD line whose archive URL repo == dep_repo cur_sha <- 40-char sha parsed from that line's archive URL if upstream_sha == cur_sha: report "name (current)" else: rewrite line: replace cur_sha with upstream_sha in BOTH the archive// path segment and the -.tar.gz filename; recompute md5 (existing download+md5 machinery) report "name -> " ``` SHA comparison is plain full-string `==` (40 hex chars both sides; `.info` is written full-SHA everywhere, so no short/long prefix handling). ### Report All tracked deps are listed, unchanged ones marked `(current)`: ``` reconcile openvino: mlas d1bc25e -> a3f9c01 onednn (current) onednn_gpu (current) flatbuffers (current) onnx (current) ittapi (current) protobuf (current) rewrote 1, md5 recomputed ``` Report shows short (7-char) SHAs; the actual rewrite uses full 40-char SHAs. ## Set-drift detection (Job B) Job A tells you your 7 dep SHAs are stale. It is blind to the **set** changing: if a new openvino version drops a bundled submodule or adds one, Job A still reports "7 current" while the SlackBuild silently breaks. Job B closes that gap. **Runs on a bump OR under `--force`** — the same gate as Job A, so `--force` always yields the complete picture. On a real version bump, `old` = the hint's VERSION before the rewrite, `new` = the version being bumped to, and the diff populates `+`/`-` change glyphs. Under `--force` with no bump, `old == new ==` the current version: the diff is empty, so the roster prints with all blank-change rows (a pure `bundled`/`(ignored)` inventory). Either way the full roster prints. **Baseline is free — no stored state.** The two versions come straight from the `--check` run (old = pre-bump hint VERSION, new = bumped-to version; equal under forced no-bump). Job B fetches `.gitmodules` at both refs and diffs the submodule **path** sets: ``` old_paths <- submodule paths in .gitmodules@old new_paths <- submodule paths in .gitmodules@new added = new_paths - old_paths removed = old_paths - new_paths ``` openvino ships ~20 submodules; the user bundles 7 and gets the other ~13 (gtest, gflags, zlib, xbyak, ...) from Slackware system packages. Job B prints a **full inventory roster**: every submodule (all ~20), not just the diff, so the user has the complete picture each time. Each row carries two independent tags: - **change** (from the old→new diff): `+` added, `-` removed, blank unchanged. - **bundle membership** (from the manifest `name=path` set): a shipped dep is plain; a non-bundled one is flagged `(ignored)`. This is what keeps the stable 13 legible instead of noise — they are shown but visibly marked as not yours. Signal is raised where change × membership intersects: - **removed ∧ bundled** → `ACTION`: a dep you ship is gone upstream. - **added ∧ not bundled** → `review`: openvino has a new submodule; you decide whether it must be bundled (a SlackBuild change). - everything else → informational (bundled-unchanged, or `(ignored)`). ### Roster construction Iterate the `@new` submodule paths for `+`/unchanged rows, then append any `@old`-only paths as `-` rows (a removed submodule is absent from `@new`, so it must be pulled from the `@old` set). Cross-reference each path against the manifest for the `(ignored)` flag and the ACTION/review signal. ### Report ``` openvino submodule inventory 2024.4.1 -> 2024.5.0: - thirdparty/onnx/onnx ACTION (bundled, removed upstream) + src/plugins/foo/thirdparty/newdep review (new, not bundled) src/plugins/intel_cpu/thirdparty/mlas bundled src/plugins/intel_cpu/thirdparty/onednn bundled src/plugins/intel_gpu/thirdparty/onednn_gpu bundled thirdparty/flatbuffers/flatbuffers bundled thirdparty/ittapi/ittapi bundled thirdparty/protobuf/protobuf bundled thirdparty/gtest/gtest (ignored) thirdparty/zlib/zlib (ignored) thirdparty/xbyak (ignored) ... (all remaining submodules) ``` The roster always prints when Job B runs (bump or `--force`), even when the set is unchanged — it is a full inventory, not a diff, so an all-`bundled`/`(ignored)`, no-`+`/`-` roster still gives the user the picture. Under forced no-bump `old == new`, so the two `.gitmodules` fetches collapse to one via the `repo@ref` cache. ### Failure handling `.gitmodules` fetch failure at either ref → skip Job B with a one-line notice, do not abort. Job A (the SHA reconcile) is independent and still runs. ## GitHub API auth Contents API is ~1 call per dep (7 for openvino) at the pinned ref. Anon GitHub API is ~60/h (per the user's standing rate-limit rule), enough for a single reconcile but fragile under bursts. mkhint reads a GitHub token from nvchecker's keyfile when present and sends it as `Authorization: Bearer ` (→ 5000/h); falls back to anon when absent. Token location: `NVCHECKER_CONFIG` (`nvchecker.toml`) has `keyfile = ""` (relative to the nvchecker config dir); that keyfile has a `github = ""` line. Parse both with the existing grep/cut style (no toml library). Absent keyfile or absent github key → anon. Job B's two `.gitmodules` fetches hit `raw.githubusercontent.com` (a plain file, not the API) and do not count against the API rate limit, so they need no token. ## Error handling - **Contents API 404** (submodule path gone at this version): report `name: submodule path not found at `, skip that dep, continue. Does not abort the run. User adjusts the manifest. - **Network / rate-limit failure**: same graceful skip the neovim reconcile already uses; report the dep as unresolved, do not abort. - **DOWNLOAD line present but no manifest dep matches its repo**: left untouched. Not every `DOWNLOAD` line is a bundled dep (the primary openvino tarball is a `DOWNLOAD` line too). This mirrors the neovim path's "extra line, leave it" behavior. - **Manifest dep whose repo matches no DOWNLOAD line**: report as an FYI (`name: no matching DOWNLOAD line`), never add a line (would need a SlackBuild change), matching the neovim "manifest dep with no hint line" FYI. ## Code shape - `load_bundle_manifests`: parse field-2 mode; dispatch `url` (existing shape) vs `sha` (new: repo, version-template, name=path map). Store mode per package. - `reconcile_bundle_deps`: dispatch on stored mode. Existing url logic factored into `reconcile_bundle_deps_url` (behavior unchanged); new `reconcile_bundle_deps_sha` for sha mode. - New helpers: `_github_token` (read nvchecker keyfile → token or empty), `_fetch_submodule_sha` (contents API call → sha + submodule repo), `_rewrite_sha_download_line` (swap sha in path+filename, recompute md5). - Job B: `_fetch_gitmodules_paths ` (fetch `.gitmodules`, echo submodule paths), `detect_set_drift

` (diff the two path sets, cross-ref manifest, print the report). Called from the `--check` bump path once the primary version is confirmed changed, before/alongside the Job A reconcile. - Per-run cache of contents-API responses keyed by `path@ref` (aligns with the existing "cache manifest per run" TODO so a dep is fetched once). `.gitmodules` fetches cached by `repo@ref` likewise. ## Test coverage Mock the contents API via the existing fake-`wget`/`curl` harness returning canned JSON (`{"sha": "...", "submodule_git_url": "...git", "type": "submodule"}`). Mock hint has full-SHA `DOWNLOAD` lines. New cases: | ID | Scenario | |----|----------| | T-SHA1 | sha-mode dep SHA differs — line rewritten (path + filename), md5 recomputed, report `old7 -> new7` | | T-SHA2 | sha-mode dep SHA equal — no change, report `(current)` | | T-SHA3 | contents API 404 for a path — dep skipped, `not found` reported, run continues, hint unchanged for that dep | | T-SHA4 | `--new` on a sha-mode package — report printed, hint unchanged | | T-SHA5 | two deps same repo different paths (onednn cpu vs onednn_gpu) — each rewritten against its own path's SHA, no cross-contamination | | T-SHA6 | github token present in keyfile — API call carries `Authorization` header (assert on fake curl's seen args); token absent — anon, no header | | T-SHA7 | manifest dep repo matches no DOWNLOAD line — FYI reported, nothing added | | T-SHA8 | neovim `url`-mode entry still reconciles after the field-2 migration (regression) | | T-SET1 | Job B: `@new` adds a submodule not in `@old`, not in manifest — roster row `+ ... review`, no ACTION | | T-SET2 | Job B: `@new` drops a submodule that was in `@old` and in the manifest — roster row `- ... ACTION` (pulled from `@old`) | | T-SET3 | Job B: set unchanged across the bump — full roster still prints, all rows blank-change, bundled/(ignored) tags correct | | T-SET4 | Job B: unbundled submodules present in both refs — appear in roster flagged `(ignored)`, never ACTION/review | | T-SET5 | Job B: `.gitmodules` fetch fails at one ref — Job B skipped with notice, Job A still runs | | T-SET6 | Job B under `--force` at unchanged version — old==new, single `.gitmodules` fetch (cached), roster prints with all blank-change rows | Follow the harness gotcha: color/grep guard asserts use `(( FAIL++ )) || true` so the first failing assertion does not abort the run under `set -e`. ## Prerequisite (user-owned, out of scope for mkhint) The user rewrites `openvino.SlackBuild` + `openvino.info` so: - every bundled dep is a `github.com///archive//-.tar.gz` `DOWNLOAD_x86_64` line, - the SlackBuild extracts/uses those tarballs by reading the hint values (no hardcoded `COMMIT_*`/`VERSION_*`), - the `openvino` sha-mode entry is added to `BUNDLE_MANIFEST_FILE`. Future openvino versions may add/remove bundled deps; the user adjusts the `.info` `DOWNLOAD` lines and the manifest `name=path` list accordingly, keeping SHA-only.