added plan to include hf_xet python3 package

1 files changed, 311 insertions, 0 deletions

diff --git a/.extras/docs/python3-hf_xet.md b/.extras/docs/python3-hf_xet.md
new file mode 100644
index 0000000..56109ca
--- /dev/null
+++ b/.extras/docs/python3-hf_xet.md
@@ -0,0 +1,311 @@
+# Adding python3-hf_xet as a separate SlackBuild
+
+Plan for packaging the `hf_xet` Rust/PyO3 extension that accelerates
+`huggingface_hub` with Xet storage (chunk-based deduplication). Research only,
+nothing has been built or committed yet.
+
+Written 2026-08-11. All upstream versions and checksums were verified on that
+date. Re-verify before acting if a new hf-xet release has shipped.
+
+---
+
+## Conclusions up front
+
+**Package it as `python3-hf_xet`, a separate package.** It is an optional
+accelerator, not a hard runtime dependency of `huggingface_hub`, so it should
+not be added to `python3-hf_xet`'s REQUIRES. Without it, `huggingface_hub`
+falls back to regular HTTP download and prints a warning, which is the
+behaviour that motivated this plan (see `TODO-AND-NOTES.md`).
+
+**The SBo name is free.** Searched slackbuilds.org for both `hf_xet` and
+`hf-xet`: no results.
+
+**Build system is maturin (PyO3).** SBo-stable already has `python3-maturin`
+1.14.1, which satisfies `maturin>=1.7,<2.0`. That package REQUIRES `rust-opt`
+(currently 1.97.1), which provides `/opt/rust/bin/cargo` and `rustc`. Rust
+edition 2024 is used, which needs rustc 1.85+, so 1.97.1 is fine.
+
+**The closest SBo precedent is `python3-orjson`** — same stack: Rust/PyO3
+extension built with maturin, same SlackBuild pattern (PATH and
+LD_LIBRARY_PATH for rust-opt, `python3 -m build --no-isolation`, `python3 -m
+installer`). The new script should follow it closely.
+
+**The sdist does not vendor cargo dependencies.** The PyPI sdist
+(`hf_xet-1.6.0.tar.gz`, 920 KB) contains the workspace source (the `hf_xet`
+crate plus sibling path-dependency crates: `xet_pkg`, `xet_runtime`,
+`xet_client`, `xet_core_structures`, `xet_data`) and `Cargo.lock` (148 KB),
+but not the crates.io dependencies. Building from sdist requires network
+access to fetch from crates.io. This is the main decision point, see below.
+
+---
+
+## What hf_xet is
+
+`hf-xet` enables `huggingface_hub` to use Xet storage for uploading and
+downloading from the HF Hub. Xet storage provides chunk-based deduplication,
+efficient storage/retrieval with local disk caching, and backwards
+compatibility with Git LFS. The library is not meant to be used directly, it
+is imported by `huggingface_hub` when available.
+
+As of `huggingface_hub` 0.32.0, `hf_xet` is installed automatically as a
+dependency. On this system, where `python3-huggingface_hub` is built from
+source without it, the CLI prints:
+
+```
+Xet Storage is enabled for this repo, but the 'hf_xet' package is not
+installed. Falling back to regular HTTP download. For better performance,
+install the package with: `pip install huggingface_hub[hf_xet]` or
+`pip install hf_xet`
+```
+
+That warning is the reason this package is wanted.
+
+### Upstream facts (verified 2026-08-11)
+
+| Item | Value |
+|---|---|
+| PyPI name | `hf-xet` |
+| Import name | `hf_xet` |
+| Latest version | `1.6.0` |
+| License | Apache-2.0 |
+| requires_python | `>=3.8` |
+| Build system | maturin (`>=1.7,<2.0`) |
+| Rust edition | 2024 (needs rustc 1.85+) |
+| PyO3 version | 0.29, abi3-py38 (one wheel for all CPython 3.8+) |
+| Runtime deps | none (only `pytest` for the `tests` extra) |
+| Homepage | `https://github.com/huggingface/xet-core` |
+| sdist | `hf_xet-1.6.0.tar.gz`, 920,527 bytes, md5 `bf1ab4c595afa39ed1391294dc1d9f01` |
+| sdist URL | `https://files.pythonhosted.org/packages/source/h/hf-xet/hf_xet-1.6.0.tar.gz` |
+
+### Workspace structure
+
+`hf_xet` lives in the `huggingface/xet-core` monorepo, excluded from the main
+workspace but depending on sibling crates via path deps:
+
+```
+xet-core/
+├── Cargo.toml          # workspace, excludes hf_xet
+├── Cargo.lock          # 148 KB, shared across the workspace
+├── hf_xet/             # the PyO3 bindings (this package)
+│   ├── Cargo.toml      # path deps to ../xet_pkg, ../xet_runtime, ../xet_client
+│   └── pyproject.toml  # maturin build backend
+├── xet_pkg/            # consolidated package, depends on the rest
+├── xet_runtime/        # async runtime, config, logging
+├── xet_client/         # HTTP client for CAS and Hub backend
+├── xet_core_structures/ # MerkleHash, metadata shards, shared structs
+└── xet_data/           # chunking, dedup, file reconstruction
+```
+
+The maturin sdist includes the `hf_xet` crate and the sibling path-dep crates,
+so it is self-contained at the workspace level. It is not self-contained at
+the crates.io level.
+
+---
+
+## Decision: online build vs vendored build
+
+This is the only real design choice. Everything else follows the `python3-orjson`
+template.
+
+### Option A: online build (recommended)
+
+Do not set `CARGO_NET_OFFLINE=true`. Cargo fetches dependencies from crates.io
+at build time. The build needs network access, which the Docker test-build
+container and the buildsystem VM both have.
+
+Pros:
+- Simplest script, fewest moving parts.
+- No vendored tarball to generate, host, or keep in sync across version bumps.
+- Matches how most Rust Python extensions are built outside SBo.
+
+Cons:
+- Not SBo-conventional. SBo packages are expected to build offline.
+- If submitting to SBo later, a vendored tarball would need to be added.
+
+### Option B: vendored build
+
+Generate a vendored cargo deps tarball via `cargo vendor`, host it (the SBo
+maintainer `fourtysixandtwo` hosts on `slackware.uk`), and add it as a second
+`DOWNLOAD`/`MD5SUM` entry. Set `CARGO_NET_OFFLINE=true` and point
+`CARGO_HOME` at the vendored directory.
+
+Pros:
+- SBo-conventional offline build.
+- No network dependency at build time.
+
+Cons:
+- The vendored tarball would be roughly 50-200 MB (the dep tree includes
+  `reqwest`, `tokio`, `pyo3`, `git2`, `blake3`, `hyper`, etc.).
+- Needs hosting somewhere.
+- Every version bump requires regenerating and re-hosting the tarball.
+- `python3-maturin` ships a `mkvendored.sh` script that could be used as a
+  reference, but the workflow is still manual.
+
+### Recommendation
+
+Start with option A (online build). It gets the package working with the
+least effort. If submitting to SBo becomes the goal, switch to option B at
+that point, the script change is small (add the second `DOWNLOAD`, set
+`CARGO_NET_OFFLINE=true`, point `CARGO_HOME` at the vendor dir).
+
+---
+
+## Plan
+
+### Step 1: create the package files
+
+Create `python3-hf_xet/` with the standard four files:
+
+```
+python3-hf_xet/
+├── python3-hf_xet.SlackBuild
+├── python3-hf_xet.info
+├── README
+└── slack-desc
+```
+
+#### SlackBuild
+
+Base the script on `python3-orjson` (`/var/lib/sbopkg/SBo-stable/python/python3-orjson/python3-orjson.SlackBuild`),
+which is the closest analog. Key points:
+
+- `PRGNAM=python3-hf_xet`, `SRCNAM=hf_xet` (the PyPI sdist name).
+- Standard ARCH detection, SLKCFLAGS, LIBDIRSUFFIX.
+- Extract: `tar xvf $CWD/$SRCNAM-$VERSION.tar.gz`.
+- Set up rust-opt:
+  ```sh
+  export PATH="/opt/rust/bin:$PATH"
+  export LD_LIBRARY_PATH="/opt/rust/lib$LIBDIRSUFFIX${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
+  ```
+- `export MATURIN_NO_INSTALL_RUST=1` — prevents maturin from auto-installing
+  rust if rust-opt is missing.
+- `export CARGO_HOME=$(pwd)/.cargo` — keeps cargo writes inside `$TMP`.
+- Do **not** set `CARGO_NET_OFFLINE=true` (online build, option A).
+- Build and install:
+  ```sh
+  python3 -m build --wheel --no-isolation
+  python3 -m installer --destdir "$PKG" dist/*.whl
+  ```
+- Strip, docs (`LICENSE README*`), slack-desc, makepkg.
+
+#### .info
+
+```
+PRGNAM="python3-hf_xet"
+VERSION="1.6.0"
+HOMEPAGE="https://github.com/huggingface/xet-core"
+DOWNLOAD="https://files.pythonhosted.org/packages/source/h/hf-xet/hf_xet-1.6.0.tar.gz"
+MD5SUM="bf1ab4c595afa39ed1391294dc1d9f01"
+DOWNLOAD_x86_64=""
+MD5SUM_x86_64=""
+REQUIRES="python3-maturin"
+MAINTAINER="danix"
+EMAIL="danix@danix.xyz"
+```
+
+`REQUIRES="python3-maturin"` because maturin is the build backend. `rust-opt`
+is pulled in transitively through `python3-maturin`'s own REQUIRES. No runtime
+REQUIRES, `hf_xet` is a standalone extension module.
+
+#### slack-desc
+
+11 lines, `python3-hf_xet:` prefix. Short description: "Rust extension for
+Hugging Face Hub Xet storage (dedup and fast transfers)."
+
+#### README
+
+Standard SBo README: what it is, what it does, that it is an optional
+accelerator for `huggingface_hub`, build deps (rust-opt, python3-maturin).
+
+### Step 2: add nvchecker entry
+
+Append to `.extras/nvchecker.toml`:
+
+```toml
+[python3-hf_xet]
+source = "pypi"
+pypi = "hf-xet"
+```
+
+### Step 3: verify and lint
+
+```sh
+cd python3-hf_xet && sbofixinfo
+cd python3-hf_xet && sbodl
+cd python3-hf_xet && sbolint
+```
+
+`sbodl` must report `md5sum matches OK`. If the checksum is stale, use the
+two-pass `sbodl` procedure from AGENTS.md.
+
+### Step 4: test build
+
+Use the `test-build-slackbuild` skill. Run both trees since this package is
+SBo-submittable:
+
+```sh
+test-build python3-hf_xet            # -current
+test-build --stable python3-hf_xet   # 15.0
+```
+
+Watch for:
+- rust-opt version on -stable (if 15.0's rust-opt is older than 1.85, edition
+  2024 will fail to compile).
+- Network access in the Docker container (cargo needs to fetch from
+  crates.io for the online build).
+- The build is slow (Rust compilation with LTO, `codegen-units = 1`).
+
+### Step 5: report and wait
+
+Present a summary of all changes and test results. Wait for commit
+instruction.
+
+---
+
+## Known problem areas
+
+**-stable rust-opt age.** The `Cargo.toml` uses `edition = "2024"`, which
+requires rustc 1.85+. If SBo-stable's `rust-opt` is older than 1.85, the
+build will fail on 15.0. SBo-stable currently ships rust-opt 1.97.1 (verified
+2026-08-11 in `/var/lib/sbopkg/SBo-stable/development/rust-opt/rust-opt.info`),
+which is fine, but this should be re-checked at build time.
+
+**Build time.** The release profile uses `lto = true` and
+`codegen-units = 1`, plus the dep tree is large (reqwest, tokio, pyo3, git2).
+Expect a multi-minute compile even on fast hardware. The Docker test-build
+timeout may need to be generous.
+
+**No runtime deps, but a transitive build-toolchain dep.** `python3-maturin`
+REQUIRES `rust-opt` and `python3-setuptools-rust-opt`. Anyone building
+`python3-hf_xet` needs the full Rust toolchain installed. This is expected
+for any maturin-built package and is not a problem specific to hf_xet.
+
+**sdist does not include vendored crates.** See the decision section above.
+The online build is the pragmatic choice for this repo. If SBo submission is
+the goal, vendoring must be added.
+
+**`huggingface_hub` integration is automatic.** Once `hf_xet` is installed,
+`huggingface_hub` detects it and uses it. No configuration needed. The
+warning in `TODO-AND-NOTES.md` disappears.
+
+---
+
+## What NOT to change
+
+**Do not add `python3-hf_xet` to `python3-huggingface_hub`'s REQUIRES.** It
+is optional. Forcing it would pull in the entire Rust toolchain as a build
+dependency of `huggingface_hub`, which is unreasonable for users who only
+want the Python API.
+
+**Do not use the prebuilt wheels.** PyPI ships manylinux wheels, but SBo
+policy is to build from source. The sdist exists and is buildable.
+
+---
+
+## Status
+
+Updated 2026-08-11.
+
+Research complete. No files created yet. The package is ready to be built
+when the user decides to proceed. The only open decision is online vs
+vendored build (recommended: online).