diff options
Diffstat (limited to '.extras/docs')
| -rw-r--r-- | .extras/docs/python3-hf_xet.md | 311 |
1 files changed, 311 insertions, 0 deletions
diff --git a/.extras/docs/python3-hf_xet.md b/.extras/docs/python3-hf_xet.md new file mode 100644 index 0000000..56109ca --- /dev/null +++ b/.extras/docs/python3-hf_xet.md @@ -0,0 +1,311 @@ +# Adding python3-hf_xet as a separate SlackBuild + +Plan for packaging the `hf_xet` Rust/PyO3 extension that accelerates +`huggingface_hub` with Xet storage (chunk-based deduplication). Research only, +nothing has been built or committed yet. + +Written 2026-08-11. All upstream versions and checksums were verified on that +date. Re-verify before acting if a new hf-xet release has shipped. + +--- + +## Conclusions up front + +**Package it as `python3-hf_xet`, a separate package.** It is an optional +accelerator, not a hard runtime dependency of `huggingface_hub`, so it should +not be added to `python3-hf_xet`'s REQUIRES. Without it, `huggingface_hub` +falls back to regular HTTP download and prints a warning, which is the +behaviour that motivated this plan (see `TODO-AND-NOTES.md`). + +**The SBo name is free.** Searched slackbuilds.org for both `hf_xet` and +`hf-xet`: no results. + +**Build system is maturin (PyO3).** SBo-stable already has `python3-maturin` +1.14.1, which satisfies `maturin>=1.7,<2.0`. That package REQUIRES `rust-opt` +(currently 1.97.1), which provides `/opt/rust/bin/cargo` and `rustc`. Rust +edition 2024 is used, which needs rustc 1.85+, so 1.97.1 is fine. + +**The closest SBo precedent is `python3-orjson`** — same stack: Rust/PyO3 +extension built with maturin, same SlackBuild pattern (PATH and +LD_LIBRARY_PATH for rust-opt, `python3 -m build --no-isolation`, `python3 -m +installer`). The new script should follow it closely. + +**The sdist does not vendor cargo dependencies.** The PyPI sdist +(`hf_xet-1.6.0.tar.gz`, 920 KB) contains the workspace source (the `hf_xet` +crate plus sibling path-dependency crates: `xet_pkg`, `xet_runtime`, +`xet_client`, `xet_core_structures`, `xet_data`) and `Cargo.lock` (148 KB), +but not the crates.io dependencies. Building from sdist requires network +access to fetch from crates.io. This is the main decision point, see below. + +--- + +## What hf_xet is + +`hf-xet` enables `huggingface_hub` to use Xet storage for uploading and +downloading from the HF Hub. Xet storage provides chunk-based deduplication, +efficient storage/retrieval with local disk caching, and backwards +compatibility with Git LFS. The library is not meant to be used directly, it +is imported by `huggingface_hub` when available. + +As of `huggingface_hub` 0.32.0, `hf_xet` is installed automatically as a +dependency. On this system, where `python3-huggingface_hub` is built from +source without it, the CLI prints: + +``` +Xet Storage is enabled for this repo, but the 'hf_xet' package is not +installed. Falling back to regular HTTP download. For better performance, +install the package with: `pip install huggingface_hub[hf_xet]` or +`pip install hf_xet` +``` + +That warning is the reason this package is wanted. + +### Upstream facts (verified 2026-08-11) + +| Item | Value | +|---|---| +| PyPI name | `hf-xet` | +| Import name | `hf_xet` | +| Latest version | `1.6.0` | +| License | Apache-2.0 | +| requires_python | `>=3.8` | +| Build system | maturin (`>=1.7,<2.0`) | +| Rust edition | 2024 (needs rustc 1.85+) | +| PyO3 version | 0.29, abi3-py38 (one wheel for all CPython 3.8+) | +| Runtime deps | none (only `pytest` for the `tests` extra) | +| Homepage | `https://github.com/huggingface/xet-core` | +| sdist | `hf_xet-1.6.0.tar.gz`, 920,527 bytes, md5 `bf1ab4c595afa39ed1391294dc1d9f01` | +| sdist URL | `https://files.pythonhosted.org/packages/source/h/hf-xet/hf_xet-1.6.0.tar.gz` | + +### Workspace structure + +`hf_xet` lives in the `huggingface/xet-core` monorepo, excluded from the main +workspace but depending on sibling crates via path deps: + +``` +xet-core/ +├── Cargo.toml # workspace, excludes hf_xet +├── Cargo.lock # 148 KB, shared across the workspace +├── hf_xet/ # the PyO3 bindings (this package) +│ ├── Cargo.toml # path deps to ../xet_pkg, ../xet_runtime, ../xet_client +│ └── pyproject.toml # maturin build backend +├── xet_pkg/ # consolidated package, depends on the rest +├── xet_runtime/ # async runtime, config, logging +├── xet_client/ # HTTP client for CAS and Hub backend +├── xet_core_structures/ # MerkleHash, metadata shards, shared structs +└── xet_data/ # chunking, dedup, file reconstruction +``` + +The maturin sdist includes the `hf_xet` crate and the sibling path-dep crates, +so it is self-contained at the workspace level. It is not self-contained at +the crates.io level. + +--- + +## Decision: online build vs vendored build + +This is the only real design choice. Everything else follows the `python3-orjson` +template. + +### Option A: online build (recommended) + +Do not set `CARGO_NET_OFFLINE=true`. Cargo fetches dependencies from crates.io +at build time. The build needs network access, which the Docker test-build +container and the buildsystem VM both have. + +Pros: +- Simplest script, fewest moving parts. +- No vendored tarball to generate, host, or keep in sync across version bumps. +- Matches how most Rust Python extensions are built outside SBo. + +Cons: +- Not SBo-conventional. SBo packages are expected to build offline. +- If submitting to SBo later, a vendored tarball would need to be added. + +### Option B: vendored build + +Generate a vendored cargo deps tarball via `cargo vendor`, host it (the SBo +maintainer `fourtysixandtwo` hosts on `slackware.uk`), and add it as a second +`DOWNLOAD`/`MD5SUM` entry. Set `CARGO_NET_OFFLINE=true` and point +`CARGO_HOME` at the vendored directory. + +Pros: +- SBo-conventional offline build. +- No network dependency at build time. + +Cons: +- The vendored tarball would be roughly 50-200 MB (the dep tree includes + `reqwest`, `tokio`, `pyo3`, `git2`, `blake3`, `hyper`, etc.). +- Needs hosting somewhere. +- Every version bump requires regenerating and re-hosting the tarball. +- `python3-maturin` ships a `mkvendored.sh` script that could be used as a + reference, but the workflow is still manual. + +### Recommendation + +Start with option A (online build). It gets the package working with the +least effort. If submitting to SBo becomes the goal, switch to option B at +that point, the script change is small (add the second `DOWNLOAD`, set +`CARGO_NET_OFFLINE=true`, point `CARGO_HOME` at the vendor dir). + +--- + +## Plan + +### Step 1: create the package files + +Create `python3-hf_xet/` with the standard four files: + +``` +python3-hf_xet/ +├── python3-hf_xet.SlackBuild +├── python3-hf_xet.info +├── README +└── slack-desc +``` + +#### SlackBuild + +Base the script on `python3-orjson` (`/var/lib/sbopkg/SBo-stable/python/python3-orjson/python3-orjson.SlackBuild`), +which is the closest analog. Key points: + +- `PRGNAM=python3-hf_xet`, `SRCNAM=hf_xet` (the PyPI sdist name). +- Standard ARCH detection, SLKCFLAGS, LIBDIRSUFFIX. +- Extract: `tar xvf $CWD/$SRCNAM-$VERSION.tar.gz`. +- Set up rust-opt: + ```sh + export PATH="/opt/rust/bin:$PATH" + export LD_LIBRARY_PATH="/opt/rust/lib$LIBDIRSUFFIX${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" + ``` +- `export MATURIN_NO_INSTALL_RUST=1` — prevents maturin from auto-installing + rust if rust-opt is missing. +- `export CARGO_HOME=$(pwd)/.cargo` — keeps cargo writes inside `$TMP`. +- Do **not** set `CARGO_NET_OFFLINE=true` (online build, option A). +- Build and install: + ```sh + python3 -m build --wheel --no-isolation + python3 -m installer --destdir "$PKG" dist/*.whl + ``` +- Strip, docs (`LICENSE README*`), slack-desc, makepkg. + +#### .info + +``` +PRGNAM="python3-hf_xet" +VERSION="1.6.0" +HOMEPAGE="https://github.com/huggingface/xet-core" +DOWNLOAD="https://files.pythonhosted.org/packages/source/h/hf-xet/hf_xet-1.6.0.tar.gz" +MD5SUM="bf1ab4c595afa39ed1391294dc1d9f01" +DOWNLOAD_x86_64="" +MD5SUM_x86_64="" +REQUIRES="python3-maturin" +MAINTAINER="danix" +EMAIL="danix@danix.xyz" +``` + +`REQUIRES="python3-maturin"` because maturin is the build backend. `rust-opt` +is pulled in transitively through `python3-maturin`'s own REQUIRES. No runtime +REQUIRES, `hf_xet` is a standalone extension module. + +#### slack-desc + +11 lines, `python3-hf_xet:` prefix. Short description: "Rust extension for +Hugging Face Hub Xet storage (dedup and fast transfers)." + +#### README + +Standard SBo README: what it is, what it does, that it is an optional +accelerator for `huggingface_hub`, build deps (rust-opt, python3-maturin). + +### Step 2: add nvchecker entry + +Append to `.extras/nvchecker.toml`: + +```toml +[python3-hf_xet] +source = "pypi" +pypi = "hf-xet" +``` + +### Step 3: verify and lint + +```sh +cd python3-hf_xet && sbofixinfo +cd python3-hf_xet && sbodl +cd python3-hf_xet && sbolint +``` + +`sbodl` must report `md5sum matches OK`. If the checksum is stale, use the +two-pass `sbodl` procedure from AGENTS.md. + +### Step 4: test build + +Use the `test-build-slackbuild` skill. Run both trees since this package is +SBo-submittable: + +```sh +test-build python3-hf_xet # -current +test-build --stable python3-hf_xet # 15.0 +``` + +Watch for: +- rust-opt version on -stable (if 15.0's rust-opt is older than 1.85, edition + 2024 will fail to compile). +- Network access in the Docker container (cargo needs to fetch from + crates.io for the online build). +- The build is slow (Rust compilation with LTO, `codegen-units = 1`). + +### Step 5: report and wait + +Present a summary of all changes and test results. Wait for commit +instruction. + +--- + +## Known problem areas + +**-stable rust-opt age.** The `Cargo.toml` uses `edition = "2024"`, which +requires rustc 1.85+. If SBo-stable's `rust-opt` is older than 1.85, the +build will fail on 15.0. SBo-stable currently ships rust-opt 1.97.1 (verified +2026-08-11 in `/var/lib/sbopkg/SBo-stable/development/rust-opt/rust-opt.info`), +which is fine, but this should be re-checked at build time. + +**Build time.** The release profile uses `lto = true` and +`codegen-units = 1`, plus the dep tree is large (reqwest, tokio, pyo3, git2). +Expect a multi-minute compile even on fast hardware. The Docker test-build +timeout may need to be generous. + +**No runtime deps, but a transitive build-toolchain dep.** `python3-maturin` +REQUIRES `rust-opt` and `python3-setuptools-rust-opt`. Anyone building +`python3-hf_xet` needs the full Rust toolchain installed. This is expected +for any maturin-built package and is not a problem specific to hf_xet. + +**sdist does not include vendored crates.** See the decision section above. +The online build is the pragmatic choice for this repo. If SBo submission is +the goal, vendoring must be added. + +**`huggingface_hub` integration is automatic.** Once `hf_xet` is installed, +`huggingface_hub` detects it and uses it. No configuration needed. The +warning in `TODO-AND-NOTES.md` disappears. + +--- + +## What NOT to change + +**Do not add `python3-hf_xet` to `python3-huggingface_hub`'s REQUIRES.** It +is optional. Forcing it would pull in the entire Rust toolchain as a build +dependency of `huggingface_hub`, which is unreasonable for users who only +want the Python API. + +**Do not use the prebuilt wheels.** PyPI ships manylinux wheels, but SBo +policy is to build from source. The sdist exists and is buildable. + +--- + +## Status + +Updated 2026-08-11. + +Research complete. No files created yet. The package is ready to be built +when the user decides to proceed. The only open decision is online vs +vendored build (recommended: online). |