path: root/AGENTS.md

# desktop-assistant

A local desktop voice assistant: speak, get a spoken answer, nothing leaves
the machine. Everything runs on already-packaged local tools; this repo is
the glue between them.

Status: **first working loop** in `assistant.py` (2026-09-25). Mic button
on/off is push-to-talk, whisper large-v3-turbo on the Arc transcribes,
the resident llama-server model answers, Kokoro speaks it. Button-off to
first spoken word ~2 s. See "Findings" and "Next steps".

Tools (2026-09-26, phase 1 of 2): the model can call read-only tools
(time, calendar, weather, search) defined as shell-command TOML files in
`~/.config/desktop-assistant/tools/`; `config.toml` there overrides the
built-in settings. Examples in `examples/`. STT, LLM and TTS stay local;
only the weather and search tools reach the network, and only when a
question needs them. Spec: `docs/superpowers/specs/2026-09-26-tools-and-config-design.md`.

Run: `python3 assistant.py` (`--test` for the self-check).

---

## Pipeline

```
mic button ──> arecord ──> whisper-server (STT) ──> llama-server (LLM) ──> Kokoro (TTS) ──> aplay
```

| Stage | Package (installed) | Binary | Notes |
|-------|---------------------|--------|-------|
| STT | `whisper.cpp` 1.9.4 | `whisper-server` (also `whisper-stream`, `whisper-cli`) | SDL2 mic capture, ffmpeg decoding, Vulkan + OpenVINO backends |
| LLM | `llama.cpp-vulkan` 0.5.0 | `llama-server` | already running as a system service, see below |
| TTS | `python3-kokoro-onnx` 0.6.1 | (Python module) | uses system `python3-onnxruntime` and `espeak-ng`, no `espeakng-loader` |

The three packages are in the `my-slackbuilds` repo
(`../slackbuilds-superrepo/my-slackbuilds/`). Package-level changes (build
options, backends, bumps) happen there, under that repo's `AGENTS.md`, not
here. `openvino` 2026.4.0 is also packaged there, solely to feed whisper.cpp.

## Hardware

- Discrete GPU: Intel Arc B580 (Battlemage). Target for both Vulkan and
  OpenVINO GPU.
- Integrated GPU: AMD Radeon 610M on a Ryzen 7 9700X, also exposed through
  Vulkan (RADV). Vulkan enumeration order is Arc first, iGPU second, llvmpipe
  last. Check device indices in the startup log before picking `-dev N`,
  the iGPU is a plausible wrong pick.

## whisper.cpp: what the package gives you

Built from ggml-org upstream 1.9.4 (SBo still ships 1.6.2, which is why it
is ours). Binaries only, libraries static: bundled ggml would clash with
llama.cpp-vulkan's.

Installed tools: `whisper-cli`, `whisper-server`, `whisper-bench`,
`whisper-quantize`, `whisper-vad-speech-segments`, `parakeet-cli`,
`parakeet-quantize`, and the SDL2 mic tools `whisper-stream` (live
transcription), `whisper-command` (voice commands), `whisper-talk-llama`,
`whisper-lsp`.

Backends compiled in: CPU, BLAS (OpenBLAS, autodetected), Vulkan
(`GGML_VULKAN`), ggml OpenVINO (`GGML_OPENVINO`, whole model through
OpenVINO), plus the older `WHISPER_OPENVINO` encoder-only path.

- Vulkan and ggml-openvino both register as GPU devices. whisper uses the
  first GPU it finds; choose with `whisper-cli -dev N`.
- `whisper-bench` has **no** device flag. Compare backends with
  `whisper-cli -dev N` on the same sample and read the timings it prints at
  exit.
- The `WHISPER_OPENVINO` encoder path needs the encoder converted to
  OpenVINO IR first (upstream README, "OpenVINO Support"). Not done yet.
  The ggml-openvino device does not need that conversion.
- OpenVINO libs are found through an rpath
  (`/usr/share/openvino/runtime/lib/intel64`); no need to source
  `setupvars.sh` at runtime. The OpenVINO GPU plugin
  (`libopenvino_intel_gpu_plugin.so`) is installed.
- Input goes through ffmpeg, so any format ffmpeg reads works.
- Speech encoders expect 16 kHz mono.

`whisper-talk-llama` already does mic, whisper, LLM in one binary, but it
loads its own model rather than talking to the running `llama-server`. Worth
reading as reference, probably not the final shape.

## llama-server: already running

A system service, `/etc/rc.d/rc.llama.cpp`, runs `llama-server` in **router
mode** on `http://localhost:8181` with an OpenAI-compatible API.

- Models are defined in `/etc/llama-server/presets.ini` (sections with
  `model =` and optional `mmproj =`), GGUF files under `/data/LLM-models/`.
- `--models-max 1`: the router keeps **one** model loaded. If the assistant
  asks for a different model than the one in use, the router swaps it out,
  under whoever else is using it (e.g. llamachat). Prefer reusing whatever
  model is loaded, or accept the reload latency knowingly.
- `/v1/models` lists models and each one's `architecture.input_modalities`.
  A model reporting `"audio"` accepts `input_audio` content parts directly,
  which is a possible shortcut that skips whisper entirely. Not evaluated.

Prior art: `../local-chat` (llamachat, PySide6) is a client for the same
router. It already records 16 kHz mono WAV and sends it as `input_audio`
to audio-capable models, and parses `presets.ini`. Reuse its findings, do
not import from it.

---

## Findings

### Models (2026-09-25)

Everything under `/data/voice-models/` (outside the repo, kept apart from
`/data/LLM-models/`):

- `whisper/ggml-base.bin`, 142 MB, multilingual. SHA1 matches upstream
  (`465707469ff3a37a2b9b8d8f89f2f99de7299dac`). Deleted 2026-09-26.
- `piper/it_IT-paola-medium.onnx` and `piper/en_US-lessac-medium.onnx`,
  61 MB each, plus their `.onnx.json`. Both output 22050 Hz. Deleted
  2026-09-26.
- `samples/it.wav`, `samples/en.wav`: 10 s, 16 kHz mono mic recordings used
  for the STT comparison. Personal voice data, never copy into the repo.

### STT backends (2026-09-25)

Same 10 s clip, `ggml-base`, `whisper-cli`:

| `-dev` | Device | Total (warm) | Result |
|--------|--------|--------------|--------|
| `-ng` | CPU, 4 threads | ~370-420 ms | correct |
| 0 | Vulkan0, Arc B580 | ~160-240 ms | correct |
| 1 | Vulkan1, iGPU | ~900 ms | correct |
| 2 | OPENVINO0 | 13.7 s | garbage |

**Decision: Arc on Vulkan, `-dev 0`, `-l auto`.**

- `-dev N` indexes GPU-type devices only, not the full device list:
  0 = Arc, 1 = iGPU, 2 = OpenVINO. An index past the last GPU silently falls
  back to CPU.
- The Arc's first run after a while costs 0.6-1.4 s (Vulkan pipeline
  warm-up), later runs are fast. A long-running process pays it once.
- `-l auto` detected both languages correctly (it, p=0.999) for ~40 ms extra
  (one more encoder pass). One setup covers Italian and English.
- ggml-openvino runs on **CPU**, not the Arc: the log says "OpenVINO: using
  device CPU" even with `GGML_OPENVINO_DEVICE=GPU`, and its output is
  garbage. Only OpenCL platform installed is Mesa `rusticl`; Intel
  compute-runtime (NEO), which the OpenVINO GPU plugin needs, is not
  installed. Likely cause, not confirmed. Decision on keeping OpenVINO in
  the package is deferred to `my-slackbuilds`.
- Every run tries to load `ggml-base-encoder-openvino.xml` (the
  `WHISPER_OPENVINO` encoder path), fails, and continues. Harmless noise.
- base accuracy is fine on full sentences, weak on short words: "ho" came
  out as "a", a spoken "hi" came out as "5". Try `ggml-small` (~488 MB) if
  that hurts short commands.

### Live mic with whisper-stream (2026-09-25)

`whisper-stream -m ggml-base.bin -l auto --step 0 --length 8000 -vth 0.6`
(VAD mode), 40 s, mixed Italian/English utterances with pauses.

- No `-dev` flag; it takes the first GPU, which is the Arc (Vulkan0). Default
  SDL capture device is the USB mic, correct.
- **Not usable as the assistant's STT entry point.** Its "VAD" is a crude
  energy check: on speech activity it re-transcribes the whole last
  `--length` window, not a segmented utterance. The same sentence comes out
  2-3 times, windows straddle two utterances in different languages (one
  output mixed an English sentence with the next Italian one), and `-l auto`
  detects per window, so mixed windows garble. A window with short words
  ("hello, hi, ciao") came out as Greek gibberish.
- Content of a clean window was mostly right ("ricordami di comprare
  latte"), so the model is fine, the segmentation is the problem.
- Conclusion: the loop must segment utterances itself (push-to-talk or a real
  VAD) and send exactly one utterance to whisper, kept warm in a
  long-running process (`whisper-server`) to avoid the Arc warm-up per call.

### Push-to-talk via the mic's mute button (2026-09-25)

The USB mic's hardware mute button toggles the ALSA control
`numid=2 'Mic Capture Switch'` on card 0 (`amixer -c 0 cget numid=2` reads
on/off, observed flipping with each press). So the button *is* the
push-to-talk: switch on = start recording, switch off = stop and send the
utterance to whisper. No keybind, no VAD.

- Event-driven, no polling: `amixer -c 0 events` prints a line on each
  control change ("Ready to listen..." then events).
- Card index 0 and numid 2 are what this machine shows now; card numbering
  can change with USB plug order, so resolve by card name, not number.
- The mic also exposes a HID keyboard device (`/dev/input/event3`, group
  `input`); not needed, the ALSA switch is enough.
- **Conflict: calls.** In a call the same button unmutes the mic for the
  call, and the assistant would record and answer the call. Decision: the
  assistant is paused while **presentation mode** is on (the user turns it on
  for calls). Presentation mode lives in the desktop status registry
  `~/bin/statusctl` (quickshell repo): `statusctl presentation get` prints
  `1`/`0`, backed by `$XDG_RUNTIME_DIR/status.presentation` (missing = off).
  Check it at the moment the switch goes on, and ignore the press if `1`.
  No watcher needed. Considered and not chosen: skipping when another app
  already records from the mic (`pactl list source-outputs`, sound server is
  PipeWire with the Pulse API); fallback if presentation mode gets forgotten.

### Push-to-talk round trip with whisper-server (2026-09-25)

`whisper-server -m ggml-base.bin -dev 0 -l auto --host 127.0.0.1 --port 8182`
(port next to llama-server's 8181), throwaway bash loop: `amixer events`
drives `arecord` on/off, clip POSTed to `/inference`.

- Works end to end. Warm server: **0.10-0.23 s** per utterance, including the
  very first request (no per-call Arc warm-up once the server is up).
- `amixer events` piped into a loop is block-buffered and delivers nothing;
  needs `stdbuf -oL amixer ...`.
- Do **not** mute the mic from software (`amixer cset numid=2 off`): tried
  at loop startup and the user found it unreliable (likely out of sync with
  the button's own state/LED). The user drives mute/unmute on the button
  only; the loop just reads the current state at start.
- `amixer -c Microphone` (the card id) works, no need for the index.
- Stopping `arecord` with SIGINT leaves the WAV header unfinalized (size
  field reads as ~67108 s). whisper decodes it anyway; do not trust the
  header for the duration.
- **Language auto-detect is the weak spot with base on live speech:** an
  English sentence came out as a Greek/Hebrew/Arabic mix, "ciao" came out as
  "Tchau" (Portuguese). Italian sentence fine, "hi" fine. The same sentences
  from the pre-recorded samples detected correctly, so short or accented
  utterances push base's detector off. Only it/en will ever be used.
- Correction: the pre-recorded English sample *also* detects as Greek
  (el p=0.88, en 0.009, it 0.075), on server and CLI, CPU and GPU alike; the
  earlier CLI run decoded English text in spite of the wrong language tag.
  base's detector does not handle this speaker's English.
- Tried and rejected for base: (1) pick it/en by `language_probabilities`
  (verbose_json): picks it for English. (2) transcribe forced `it` and forced
  `en`, keep higher token-weighted `avg_logprob`: forced-it turns English into
  plausible Italian and scores *higher* (-0.164 vs -0.287). Moving to
  `ggml-small`.
- `ggml-small.bin` (466 MB, SHA1 `55356645c2b361a969dfd0ef2c5a50d530afd8d5`,
  matches upstream) in `/data/voice-models/whisper/` (deleted 2026-09-26). On the Arc via
  whisper-server: **~0.35 s per pass** on a 10 s clip. Transcripts clean on
  both samples (Italian and English self-introductions, all words right).
- small's detector *still* mislabels the English sample (la 0.45, it 0.31,
  en 0.04) though it decodes English text under auto. Not trustworthy.
- **Picker works with small:** forced `it` vs forced `en`, keep higher
  token-weighted `avg_logprob`. it.wav: it -0.196 vs en -0.426; en.wav:
  en -0.127 vs it -0.243. Both correct, ~0.5 s for the two passes.
  **STT decision: ggml-small + it/en picker.**
- Live push-to-talk with small + picker, 5 utterances: Italian sentence,
  English sentence, "hi", and a mixed it/en sentence all transcribed
  perfectly, 0.42-0.54 s each (two passes). "ciao" failed: forced `en`
  produced Chinese characters ("早, 早!") with the *better* logprob (-0.032
  vs -0.312); forced `it` gave "Ciao!". An initial `prompt` ("Ciao. Hi.")
  does not fix forced-en.
- **Fix: reject any candidate containing non-Latin letters** (it and en are
  both Latin script; `unicodedata.name(c).startswith("LATIN")` for every
  alphabetic char). With that guard all 5 live clips come out right.
- Mixed-language sentences: the picker chooses one language, but small still
  transcribes both halves correctly under either forced language.
- In the full loop the picker still failed: spoken English "What does HTML
  stand for?" -> picked it ("Che stanno per HTML?", it -0.153 vs en -0.214)
  though small's own detector said en 0.926 on that clip. Combining
  `logprob + w*log(p_detect)` over 8 saved clips: 7/8 at every w, never 8/8.
  small's two signals are each unreliable on this speaker, in different clips.

### large-v3-turbo (2026-09-25, **chosen**, replaces small)

`ggml-large-v3-turbo-q5_0.bin`, 548 MB, sha256 matches huggingface
(`394221709cd5ad1f40c46e6031ca61bce88931e6e088c188294c6d5a55ffa7e2`).

- Auto-detect correct on **all 7** single-language clips, including the
  English ones small's detector called Latin/Greek; the mixed it/en clip
  comes out "it". Transcripts good (spoken "HTML stands for" kept English).
- Its forced-language logprob picker is *worse* than small's (it.wav picks en,
  -0.225 vs -0.667): with turbo, trust the detector, drop the picker.
- Cost on the Arc: ~0.95 s (whisper-cli) / ~1.2 s (whisper-server) per
  clip. Encoder runs twice (~400 ms each): once for language detection, once
  for transcription.
- `-ac` (audio context) does not pay: 768 saves ~0.25 s but starts
  repeating words, 512 repeats more, 384 is garbage. Keep 0 (full).

### The loop, `assistant.py` (2026-09-25)

- STT: one `language=auto` pass on turbo; the answer is whichever of it/en
  the detector rates higher. Forced second pass only if detection lands on
  a third language or the text is non-Latin. whisper-server's default
  language is **en**, not auto: always send `language=auto` explicitly.
- Live: turbo still tagged one English question "it" ("What is
  slackware?"), but the text stayed English, so no harm, because:
- **TTS voice follows the reply's language, not the user's.** The model may
  answer in the other language (seen: user tagged en, reply Italian, read
  by the English voice). Per-sentence stopword count picks it/en.
- **Pad 0.5 s of silence** after each clip: whisper dropped the final word
  ("... a cosa serve [python]") when the button was pressed mid-word.
  Fixed by the padding. Recording is raw PCM (`arecord -t raw`), wrapped
  into a WAV with stdlib `wave`, which also fixes the SIGINT header problem.
- LLM: model = first one in `/v1/models` whose status is not `unloaded`
  (resident is Gemma4-12B, status `sleeping` when idle). Warm first token
  ~0.3 s; after `sleep-idle-seconds 1800` the first request waits ~45 s for
  the reload. `chat_template_kwargs.enable_thinking=false` sent; only
  `delta.content` is read.
- Timings live: STT ~1.23 s, first LLM sentence 0.6-0.8 s.
- Child processes (whisper-server, `amixer events`) get
  `prctl(PR_SET_PDEATHSIG, SIGTERM)` via ctypes in `preexec_fn`: they die
  with the assistant even on SIGKILL or a kill during startup (tested).
  Python's default SIGTERM skips `finally`; a handler calls `sys.exit`.
- Gemma confidently gave a wrong fact (a city-to-city distance off by
  ~10x). No tools yet, so no time/weather/facts lookup.
- Recordings live in `$XDG_RUNTIME_DIR` and are deleted on exit.
- The reply's first word was often clipped: each reply opens a new `aplay`
  stream and the idle sink needs a moment to wake. 0.3 s of leading silence
  per reply fixed it (confirmed by ear).

### Tools, live test (2026-09-26)

Gemma4-12B through llama-server `--jinja`, streamed OpenAI `tool_calls`.
Eight spoken questions (time, calendar, home weather, remote forecast,
distance, a person, next appointment twice).

- Tool choice and arguments were right every time. Weather for a small
  town came with the country code the parameter description asks for.
- Bare city names are ambiguous for OpenWeatherMap: "Rome" and "Roma"
  both resolve to Rome, US. The place parameter asks for `City,CC` and
  the output names the country.
- **The model has no clock.** Asked for the next appointment it called the
  calendar with a 2025 start date, then recovered by calling `now` (three
  rounds, 4.0 s). With the date and time in the system prompt every turn:
  one round, 2.0 s.
- **It gets weekdays wrong from dates** ("mercoledi 29" for a Tuesday).
  `agenda.py` prints the weekday itself; right since.
- It searches for general knowledge too (a question about a well-known
  person), because the search description says "anything you are not
  sure about". Costs ~3 s (first sentence 6.7 s). **Kept on purpose** by
  the user: facts over speed.
- First sentence with one tool: 1.9-2.2 s for local tools, ~4 s with a
  search, 6.7 s for a search whose results the model read at length.
- Announcements heard on every slow tool, no audio glitch while the
  stream waits for the tool. Straight into the answer it sounded rushed;
  a 0.6 s pause after it fixed that (by ear).
- `agenda.py` matched khal's listing over 7 and 60 days on the real vdir,
  runs in ~60 ms.

### TTS (2026-09-25)

`piper-tts` on CPU (onnxruntime), ~5 s sentence per voice:

- CLI to WAV: 0.64-0.76 s wall for ~5 s of audio, ~7x real time.
- CLI `--output-raw`, time to first byte: ~0.58 s, nearly all voice load
  (0.46 s). Python + `import piper` alone is 0.09 s.
- Voice preloaded in-process (`PiperVoice.load` once, then
  `voice.synthesize(text)` yields chunks per sentence): first chunk in
  0.035-0.069 s.
- So: a per-reply `piper-tts` process costs ~0.5 s of load each time; a
  long-running process holding the voice makes TTS latency negligible.
- `--output-raw` is 16-bit mono PCM at the voice's rate (22050 Hz):
  `piper-tts -m <voice> --output-raw | aplay -r 22050 -f S16_LE -c 1`.
- **Quality rejected by ear** on a ~20 s paragraph: Italian `paola-medium`
  flat with odd rhythm, English `lessac-medium` metallic and robotic.
  Piper speed is fine, its voices are not. Looking at alternatives.

### Kokoro-82M (2026-09-25, **chosen**)

`kokoro-onnx` 0.6.1 in a throwaway venv (`--system-site-packages`, so it
uses the system onnxruntime 1.30, CPU). Model `kokoro-v1.0.onnx` (311 MB)
and `voices-v1.0.bin` (27 MB) in `/data/voice-models/kokoro/`, from the
kokoro-onnx GitHub release `model-files-v1.0`.

- The PyPI `espeakng-loader` wheel ships a lib with a hardcoded CI data path
  and fails ("Error processing file .../phontab"). Point it at the system
  espeak-ng instead:
  `EspeakConfig(lib_path="/usr/lib64/libespeak-ng.so.1", data_path="/usr/share/espeak-ng-data")`.
  A system package would do the same and drop `espeakng-loader`.
  Done 2026-09-26: `python3-kokoro-onnx` (with `python3-phonemizer`,
  `python3-dlinfo`) finds system espeak-ng with the default config.
- Load 0.30 s. `create()` on the ~20 s paragraph: ~2.6 s, ~7x real time,
  same for `if_sara`, `im_nicola` (lang `it`), `af_heart` (`en-us`),
  `bm_george` (`en-gb`). `create()` returns only after the whole text, so
  a reply needs per-sentence synthesis (or `create_stream`) to start
  speaking early.
- **By ear: much better than piper. Kokoro chosen.** Voices: `if_sara`
  (Italian) and `af_heart` (English), both female for consistency.
  `bm_george` also liked (male, "Jarvis"-like) but not used. `if_sara` wants
  a bit more color/expressiveness.
- Kokoro's only knob is `speed`; color comes from blending voice style
  vectors (`k.get_voice_style(name)` returns an array, `create(voice=...)`
  accepts it). Compared by ear: sara alone, sara 80/heart 20, sara 65/heart
  35, sara 80/bella 20. **Italian voice chosen: `0.8*if_sara + 0.2*af_bella`,
  lang `it`.**

---

## Next steps

Steps 1-5 of the original plan are done (models, backend comparison, live
mic, TTS, glue); push-to-talk came for free from the mic button. Open, in
no particular order:

- **Run as a service/autostart** (it currently runs by hand).
- **Barge-in:** pressing the button while it speaks does not stop the
  reply yet.
- **Check presentation mode during games:** `statusctl presentation get`
  reads the stored flag; confirm it is 1 while a game holds it on.
- **Latency:** STT ~1.2 s is the biggest share; the double encoder pass
  (detect + transcribe) is the target if it matters.
- **Phase 2 tools:** tools that write or modify (events, messages,
  desktop), own spec, after phase 1 has been used for a while.
- OpenVINO in the whisper.cpp package: runs on CPU only here (no Intel
  compute-runtime), decision goes to `my-slackbuilds`.

Open questions: whether an audio-capable LLM makes whisper unnecessary.
License: GPLv2 only (chosen 2026-09-25).

## Package-side leftovers (tracked in my-slackbuilds, not here)

- whisper.cpp not yet built on the buildsystem VM via slackrepo (the
  installed package is from the local build).
- `sbopkglint` on the openvino package not done.
- OpenCL-CLHPP 2026.05.29 (needed by openvino and ggml-openvino) exists
  only as a buildsystem hint; the Docker test-build needs a temporary local
  copy when its cache is cold. Permanent fix undecided.
- whisper.cpp and openvino cannot build on Slackware 15.0 (CMake 3.26,
  glslc, old CLHPP). -current only.

---

## Agent rules

- Ask before acting when anything is ambiguous, and before committing.
- Package changes belong in `my-slackbuilds`, not here.
- Never download models or voices without saying which and how big.
- Everything stays local: no cloud STT/TTS/LLM fallbacks.