diff options
Diffstat (limited to 'AGENTS.md')
| -rw-r--r-- | AGENTS.md | 397 |
1 files changed, 397 insertions, 0 deletions
diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..899fd86 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,397 @@ +# desktop-assistant + +A local desktop voice assistant: speak, get a spoken answer, nothing leaves +the machine. Everything runs on already-packaged local tools; this repo is +the glue between them. + +Status: **first working loop** in `assistant.py` (2026-09-25). Mic button +on/off is push-to-talk, whisper large-v3-turbo on the Arc transcribes, +the resident llama-server model answers, Kokoro speaks it. Button-off to +first spoken word ~2 s. Kokoro is not packaged yet: it runs from the +repo's gitignored `.venv`. See "Findings" and "Next steps". + +Run: `.venv/bin/python assistant.py` (`--test` for the self-check). The +venv is `python3 -m venv --system-site-packages .venv && .venv/bin/pip +install kokoro-onnx`; system-site-packages gives it the system onnxruntime. + +--- + +## Pipeline + +``` +mic button ──> arecord ──> whisper-server (STT) ──> llama-server (LLM) ──> Kokoro (TTS) ──> aplay +``` + +| Stage | Package (installed) | Binary | Notes | +|-------|---------------------|--------|-------| +| STT | `whisper.cpp` 1.9.4 | `whisper-server` (also `whisper-stream`, `whisper-cli`) | SDL2 mic capture, ffmpeg decoding, Vulkan + OpenVINO backends | +| LLM | `llama.cpp-vulkan` 0.5.0 | `llama-server` | already running as a system service, see below | +| TTS | none yet: `kokoro-onnx` from PyPI in `.venv` | | uses system `python3-onnxruntime` and `espeak-ng` | +| (TTS, unused) | `piper-tts` 1.7.0 | `piper-tts` | voices rejected by ear, see Findings | + +The three packages are in the `my-slackbuilds` repo +(`../slackbuilds-superrepo/my-slackbuilds/`). Package-level changes (build +options, backends, bumps) happen there, under that repo's `AGENTS.md`, not +here. `openvino` 2026.4.0 is also packaged there, solely to feed whisper.cpp. + +## Hardware + +- Discrete GPU: Intel Arc B580 (Battlemage). Target for both Vulkan and + OpenVINO GPU. +- Integrated GPU: AMD Radeon 610M on a Ryzen 7 9700X, also exposed through + Vulkan (RADV). Vulkan enumeration order is Arc first, iGPU second, llvmpipe + last. Check device indices in the startup log before picking `-dev N`, + the iGPU is a plausible wrong pick. + +## whisper.cpp: what the package gives you + +Built from ggml-org upstream 1.9.4 (SBo still ships 1.6.2, which is why it +is ours). Binaries only, libraries static: bundled ggml would clash with +llama.cpp-vulkan's. + +Installed tools: `whisper-cli`, `whisper-server`, `whisper-bench`, +`whisper-quantize`, `whisper-vad-speech-segments`, `parakeet-cli`, +`parakeet-quantize`, and the SDL2 mic tools `whisper-stream` (live +transcription), `whisper-command` (voice commands), `whisper-talk-llama`, +`whisper-lsp`. + +Backends compiled in: CPU, BLAS (OpenBLAS, autodetected), Vulkan +(`GGML_VULKAN`), ggml OpenVINO (`GGML_OPENVINO`, whole model through +OpenVINO), plus the older `WHISPER_OPENVINO` encoder-only path. + +- Vulkan and ggml-openvino both register as GPU devices. whisper uses the + first GPU it finds; choose with `whisper-cli -dev N`. +- `whisper-bench` has **no** device flag. Compare backends with + `whisper-cli -dev N` on the same sample and read the timings it prints at + exit. +- The `WHISPER_OPENVINO` encoder path needs the encoder converted to + OpenVINO IR first (upstream README, "OpenVINO Support"). Not done yet. + The ggml-openvino device does not need that conversion. +- OpenVINO libs are found through an rpath + (`/usr/share/openvino/runtime/lib/intel64`); no need to source + `setupvars.sh` at runtime. The OpenVINO GPU plugin + (`libopenvino_intel_gpu_plugin.so`) is installed. +- Input goes through ffmpeg, so any format ffmpeg reads works. +- Speech encoders expect 16 kHz mono. + +`whisper-talk-llama` already does mic, whisper, LLM in one binary, but it +loads its own model rather than talking to the running `llama-server`. Worth +reading as reference, probably not the final shape. + +## llama-server: already running + +A system service, `/etc/rc.d/rc.llama.cpp`, runs `llama-server` in **router +mode** on `http://localhost:8181` with an OpenAI-compatible API. + +- Models are defined in `/etc/llama-server/presets.ini` (sections with + `model =` and optional `mmproj =`), GGUF files under `/data/LLM-models/`. +- `--models-max 1`: the router keeps **one** model loaded. If the assistant + asks for a different model than the one in use, the router swaps it out, + under whoever else is using it (e.g. llamachat). Prefer reusing whatever + model is loaded, or accept the reload latency knowingly. +- `/v1/models` lists models and each one's `architecture.input_modalities`. + A model reporting `"audio"` accepts `input_audio` content parts directly, + which is a possible shortcut that skips whisper entirely. Not evaluated. + +Prior art: `../local-chat` (llamachat, PySide6) is a client for the same +router. It already records 16 kHz mono WAV and sends it as `input_audio` +to audio-capable models, and parses `presets.ini`. Reuse its findings, do +not import from it. + +## piper-tts + +Prebuilt upstream wheel (OHF-voice/piper1-gpl), console script `piper-tts`, +Python module `piper` under the system python3.12. Runtime deps +`python3-onnxruntime` and `python3-pathvalidate` are packaged separately. +Voices are ONNX models plus a `.onnx.json` config; none downloaded yet. + +--- + +## Findings + +### Models (2026-09-25) + +Everything under `/data/voice-models/` (outside the repo, kept apart from +`/data/LLM-models/`): + +- `whisper/ggml-base.bin`, 142 MB, multilingual. SHA1 matches upstream + (`465707469ff3a37a2b9b8d8f89f2f99de7299dac`). +- `piper/it_IT-paola-medium.onnx` and `piper/en_US-lessac-medium.onnx`, + 61 MB each, plus their `.onnx.json`. Both output 22050 Hz. +- `samples/it.wav`, `samples/en.wav`: 10 s, 16 kHz mono mic recordings used + for the STT comparison. Personal voice data, never copy into the repo. + +### STT backends (2026-09-25) + +Same 10 s clip, `ggml-base`, `whisper-cli`: + +| `-dev` | Device | Total (warm) | Result | +|--------|--------|--------------|--------| +| `-ng` | CPU, 4 threads | ~370-420 ms | correct | +| 0 | Vulkan0, Arc B580 | ~160-240 ms | correct | +| 1 | Vulkan1, iGPU | ~900 ms | correct | +| 2 | OPENVINO0 | 13.7 s | garbage | + +**Decision: Arc on Vulkan, `-dev 0`, `-l auto`.** + +- `-dev N` indexes GPU-type devices only, not the full device list: + 0 = Arc, 1 = iGPU, 2 = OpenVINO. An index past the last GPU silently falls + back to CPU. +- The Arc's first run after a while costs 0.6-1.4 s (Vulkan pipeline + warm-up), later runs are fast. A long-running process pays it once. +- `-l auto` detected both languages correctly (it, p=0.999) for ~40 ms extra + (one more encoder pass). One setup covers Italian and English. +- ggml-openvino runs on **CPU**, not the Arc: the log says "OpenVINO: using + device CPU" even with `GGML_OPENVINO_DEVICE=GPU`, and its output is + garbage. Only OpenCL platform installed is Mesa `rusticl`; Intel + compute-runtime (NEO), which the OpenVINO GPU plugin needs, is not + installed. Likely cause, not confirmed. Decision on keeping OpenVINO in + the package is deferred to `my-slackbuilds`. +- Every run tries to load `ggml-base-encoder-openvino.xml` (the + `WHISPER_OPENVINO` encoder path), fails, and continues. Harmless noise. +- base accuracy is fine on full sentences, weak on short words: "ho" came + out as "a", a spoken "hi" came out as "5". Try `ggml-small` (~488 MB) if + that hurts short commands. + +### Live mic with whisper-stream (2026-09-25) + +`whisper-stream -m ggml-base.bin -l auto --step 0 --length 8000 -vth 0.6` +(VAD mode), 40 s, mixed Italian/English utterances with pauses. + +- No `-dev` flag; it takes the first GPU, which is the Arc (Vulkan0). Default + SDL capture device is the USB mic, correct. +- **Not usable as the assistant's STT entry point.** Its "VAD" is a crude + energy check: on speech activity it re-transcribes the whole last + `--length` window, not a segmented utterance. The same sentence comes out + 2-3 times, windows straddle two utterances in different languages (one + output mixed an English sentence with the next Italian one), and `-l auto` + detects per window, so mixed windows garble. A window with short words + ("hello, hi, ciao") came out as Greek gibberish. +- Content of a clean window was mostly right ("ricordami di comprare + latte"), so the model is fine, the segmentation is the problem. +- Conclusion: the loop must segment utterances itself (push-to-talk or a real + VAD) and send exactly one utterance to whisper, kept warm in a + long-running process (`whisper-server`) to avoid the Arc warm-up per call. + +### Push-to-talk via the mic's mute button (2026-09-25) + +The USB mic's hardware mute button toggles the ALSA control +`numid=2 'Mic Capture Switch'` on card 0 (`amixer -c 0 cget numid=2` reads +on/off, observed flipping with each press). So the button *is* the +push-to-talk: switch on = start recording, switch off = stop and send the +utterance to whisper. No keybind, no VAD. + +- Event-driven, no polling: `amixer -c 0 events` prints a line on each + control change ("Ready to listen..." then events). +- Card index 0 and numid 2 are what this machine shows now; card numbering + can change with USB plug order, so resolve by card name, not number. +- The mic also exposes a HID keyboard device (`/dev/input/event3`, group + `input`); not needed, the ALSA switch is enough. +- **Conflict: calls.** In a call the same button unmutes the mic for the + call, and the assistant would record and answer the call. Decision: the + assistant is paused while **presentation mode** is on (the user turns it on + for calls). Presentation mode lives in the desktop status registry + `~/bin/statusctl` (quickshell repo): `statusctl presentation get` prints + `1`/`0`, backed by `$XDG_RUNTIME_DIR/status.presentation` (missing = off). + Check it at the moment the switch goes on, and ignore the press if `1`. + No watcher needed. Considered and not chosen: skipping when another app + already records from the mic (`pactl list source-outputs`, sound server is + PipeWire with the Pulse API); fallback if presentation mode gets forgotten. + +### Push-to-talk round trip with whisper-server (2026-09-25) + +`whisper-server -m ggml-base.bin -dev 0 -l auto --host 127.0.0.1 --port 8182` +(port next to llama-server's 8181), throwaway bash loop: `amixer events` +drives `arecord` on/off, clip POSTed to `/inference`. + +- Works end to end. Warm server: **0.10-0.23 s** per utterance, including the + very first request (no per-call Arc warm-up once the server is up). +- `amixer events` piped into a loop is block-buffered and delivers nothing; + needs `stdbuf -oL amixer ...`. +- Do **not** mute the mic from software (`amixer cset numid=2 off`): tried + at loop startup and the user found it unreliable (likely out of sync with + the button's own state/LED). The user drives mute/unmute on the button + only; the loop just reads the current state at start. +- `amixer -c Microphone` (the card id) works, no need for the index. +- Stopping `arecord` with SIGINT leaves the WAV header unfinalized (size + field reads as ~67108 s). whisper decodes it anyway; do not trust the + header for the duration. +- **Language auto-detect is the weak spot with base on live speech:** an + English sentence came out as a Greek/Hebrew/Arabic mix, "ciao" came out as + "Tchau" (Portuguese). Italian sentence fine, "hi" fine. The same sentences + from the pre-recorded samples detected correctly, so short or accented + utterances push base's detector off. Only it/en will ever be used. +- Correction: the pre-recorded English sample *also* detects as Greek + (el p=0.88, en 0.009, it 0.075), on server and CLI, CPU and GPU alike; the + earlier CLI run decoded English text in spite of the wrong language tag. + base's detector does not handle this speaker's English. +- Tried and rejected for base: (1) pick it/en by `language_probabilities` + (verbose_json): picks it for English. (2) transcribe forced `it` and forced + `en`, keep higher token-weighted `avg_logprob`: forced-it turns English into + plausible Italian and scores *higher* (-0.164 vs -0.287). Moving to + `ggml-small`. +- `ggml-small.bin` (466 MB, SHA1 `55356645c2b361a969dfd0ef2c5a50d530afd8d5`, + matches upstream) in `/data/voice-models/whisper/`. On the Arc via + whisper-server: **~0.35 s per pass** on a 10 s clip. Transcripts clean on + both samples (Italian and English self-introductions, all words right). +- small's detector *still* mislabels the English sample (la 0.45, it 0.31, + en 0.04) though it decodes English text under auto. Not trustworthy. +- **Picker works with small:** forced `it` vs forced `en`, keep higher + token-weighted `avg_logprob`. it.wav: it -0.196 vs en -0.426; en.wav: + en -0.127 vs it -0.243. Both correct, ~0.5 s for the two passes. + **STT decision: ggml-small + it/en picker.** +- Live push-to-talk with small + picker, 5 utterances: Italian sentence, + English sentence, "hi", and a mixed it/en sentence all transcribed + perfectly, 0.42-0.54 s each (two passes). "ciao" failed: forced `en` + produced Chinese characters ("早, 早!") with the *better* logprob (-0.032 + vs -0.312); forced `it` gave "Ciao!". An initial `prompt` ("Ciao. Hi.") + does not fix forced-en. +- **Fix: reject any candidate containing non-Latin letters** (it and en are + both Latin script; `unicodedata.name(c).startswith("LATIN")` for every + alphabetic char). With that guard all 5 live clips come out right. +- Mixed-language sentences: the picker chooses one language, but small still + transcribes both halves correctly under either forced language. +- In the full loop the picker still failed: spoken English "What does HTML + stand for?" -> picked it ("Che stanno per HTML?", it -0.153 vs en -0.214) + though small's own detector said en 0.926 on that clip. Combining + `logprob + w*log(p_detect)` over 8 saved clips: 7/8 at every w, never 8/8. + small's two signals are each unreliable on this speaker, in different clips. + +### large-v3-turbo (2026-09-25, **chosen**, replaces small) + +`ggml-large-v3-turbo-q5_0.bin`, 548 MB, sha256 matches huggingface +(`394221709cd5ad1f40c46e6031ca61bce88931e6e088c188294c6d5a55ffa7e2`). + +- Auto-detect correct on **all 7** single-language clips, including the + English ones small's detector called Latin/Greek; the mixed it/en clip + comes out "it". Transcripts good (spoken "HTML stands for" kept English). +- Its forced-language logprob picker is *worse* than small's (it.wav picks en, + -0.225 vs -0.667): with turbo, trust the detector, drop the picker. +- Cost on the Arc: ~0.95 s (whisper-cli) / ~1.2 s (whisper-server) per + clip. Encoder runs twice (~400 ms each): once for language detection, once + for transcription. +- `-ac` (audio context) does not pay: 768 saves ~0.25 s but starts + repeating words, 512 repeats more, 384 is garbage. Keep 0 (full). + +### The loop, `assistant.py` (2026-09-25) + +- STT: one `language=auto` pass on turbo; the answer is whichever of it/en + the detector rates higher. Forced second pass only if detection lands on + a third language or the text is non-Latin. whisper-server's default + language is **en**, not auto: always send `language=auto` explicitly. +- Live: turbo still tagged one English question "it" ("What is + slackware?"), but the text stayed English, so no harm, because: +- **TTS voice follows the reply's language, not the user's.** The model may + answer in the other language (seen: user tagged en, reply Italian, read + by the English voice). Per-sentence stopword count picks it/en. +- **Pad 0.5 s of silence** after each clip: whisper dropped the final word + ("... a cosa serve [python]") when the button was pressed mid-word. + Fixed by the padding. Recording is raw PCM (`arecord -t raw`), wrapped + into a WAV with stdlib `wave`, which also fixes the SIGINT header problem. +- LLM: model = first one in `/v1/models` whose status is not `unloaded` + (resident is Gemma4-12B, status `sleeping` when idle). Warm first token + ~0.3 s; after `sleep-idle-seconds 1800` the first request waits ~45 s for + the reload. `chat_template_kwargs.enable_thinking=false` sent; only + `delta.content` is read. +- Timings live: STT ~1.23 s, first LLM sentence 0.6-0.8 s. +- Child processes (whisper-server, `amixer events`) get + `prctl(PR_SET_PDEATHSIG, SIGTERM)` via ctypes in `preexec_fn`: they die + with the assistant even on SIGKILL or a kill during startup (tested). + Python's default SIGTERM skips `finally`; a handler calls `sys.exit`. +- Gemma confidently gave a wrong fact (a city-to-city distance off by + ~10x). No tools yet, so no time/weather/facts lookup. +- Recordings live in `$XDG_RUNTIME_DIR` and are deleted on exit. +- The reply's first word was often clipped: each reply opens a new `aplay` + stream and the idle sink needs a moment to wake. 0.3 s of leading silence + per reply fixed it (confirmed by ear). + +### TTS (2026-09-25) + +`piper-tts` on CPU (onnxruntime), ~5 s sentence per voice: + +- CLI to WAV: 0.64-0.76 s wall for ~5 s of audio, ~7x real time. +- CLI `--output-raw`, time to first byte: ~0.58 s, nearly all voice load + (0.46 s). Python + `import piper` alone is 0.09 s. +- Voice preloaded in-process (`PiperVoice.load` once, then + `voice.synthesize(text)` yields chunks per sentence): first chunk in + 0.035-0.069 s. +- So: a per-reply `piper-tts` process costs ~0.5 s of load each time; a + long-running process holding the voice makes TTS latency negligible. +- `--output-raw` is 16-bit mono PCM at the voice's rate (22050 Hz): + `piper-tts -m <voice> --output-raw | aplay -r 22050 -f S16_LE -c 1`. +- **Quality rejected by ear** on a ~20 s paragraph: Italian `paola-medium` + flat with odd rhythm, English `lessac-medium` metallic and robotic. + Piper speed is fine, its voices are not. Looking at alternatives. + +### Kokoro-82M (2026-09-25, **chosen**) + +`kokoro-onnx` 0.6.1 in a throwaway venv (`--system-site-packages`, so it +uses the system onnxruntime 1.30, CPU). Model `kokoro-v1.0.onnx` (311 MB) +and `voices-v1.0.bin` (27 MB) in `/data/voice-models/kokoro/`, from the +kokoro-onnx GitHub release `model-files-v1.0`. + +- The PyPI `espeakng-loader` wheel ships a lib with a hardcoded CI data path + and fails ("Error processing file .../phontab"). Point it at the system + espeak-ng instead: + `EspeakConfig(lib_path="/usr/lib64/libespeak-ng.so.1", data_path="/usr/share/espeak-ng-data")`. + A system package would do the same and drop `espeakng-loader`. +- Load 0.30 s. `create()` on the ~20 s paragraph: ~2.6 s, ~7x real time, + same for `if_sara`, `im_nicola` (lang `it`), `af_heart` (`en-us`), + `bm_george` (`en-gb`). `create()` returns only after the whole text, so + a reply needs per-sentence synthesis (or `create_stream`) to start + speaking early. +- **By ear: much better than piper. Kokoro chosen.** Voices: `if_sara` + (Italian) and `af_heart` (English), both female for consistency. + `bm_george` also liked (male, "Jarvis"-like) but not used. `if_sara` wants + a bit more color/expressiveness. +- Kokoro's only knob is `speed`; color comes from blending voice style + vectors (`k.get_voice_style(name)` returns an array, `create(voice=...)` + accepts it). Compared by ear: sara alone, sara 80/heart 20, sara 65/heart + 35, sara 80/bella 20. **Italian voice chosen: `0.8*if_sara + 0.2*af_bella`, + lang `it`.** + +--- + +## Next steps + +Steps 1-5 of the original plan are done (models, backend comparison, live +mic, TTS, glue); push-to-talk came for free from the mic button. Open, in +no particular order: + +- **Packaging Kokoro** in `my-slackbuilds` (`kokoro-onnx` without + `espeakng-loader`, pointed at system espeak-ng), then drop `.venv`. +- **Run as a service/autostart** (it currently runs by hand). +- **Barge-in:** pressing the button while it speaks does not stop the + reply yet. +- **Check presentation mode during games:** `statusctl presentation get` + reads the stored flag; confirm it is 1 while a game holds it on. +- **Latency:** STT ~1.2 s is the biggest share; the double encoder pass + (detect + transcribe) is the target if it matters. +- **Tools** (time, weather, calendar) so the model stops guessing facts. +- Piper leftovers: package unused, voices in `/data/voice-models/piper/` + (~120 MB) can go. ggml-base and ggml-small too, once turbo is settled. +- OpenVINO in the whisper.cpp package: runs on CPU only here (no Intel + compute-runtime), decision goes to `my-slackbuilds`. + +Open questions: whether an audio-capable LLM makes whisper unnecessary. +License: GPLv2 only (chosen 2026-09-25). + +## Package-side leftovers (tracked in my-slackbuilds, not here) + +- whisper.cpp not yet built on the buildsystem VM via slackrepo (the + installed package is from the local build). +- `sbopkglint` on the openvino package not done. +- OpenCL-CLHPP 2026.05.29 (needed by openvino and ggml-openvino) exists + only as a buildsystem hint; the Docker test-build needs a temporary local + copy when its cache is cold. Permanent fix undecided. +- whisper.cpp and openvino cannot build on Slackware 15.0 (CMake 3.26, + glslc, old CLHPP). -current only. + +--- + +## Agent rules + +- Ask before acting when anything is ambiguous, and before committing. +- Package changes belong in `my-slackbuilds`, not here. +- Never download models or voices without saying which and how big. +- Everything stays local: no cloud STT/TTS/LLM fallbacks. |