path: root/AGENTS.md

# desktop-assistant

A local text assistant daemon: questions in and streamed replies out on a
Unix socket, answered by the model already loaded in llama-server, with
read-only tools and a SQLite history. Nothing leaves the machine except
what an enabled tool fetches.

Status (2026-10-16): **text daemon**. The front end is the quickshell
canvas (`desktop/assistant/` in the quickshell repo, SUPER+Home). Spec:
`docs/superpowers/specs/2026-10-16-text-daemon-design.md`.

The voice version is the tag `voice-v1`: push-to-talk on the mic's mute
button, whisper large-v3-turbo, Kokoro. These sections describe only that
version, and are kept because a return to voice starts from them:
"Hardware", "whisper.cpp", and under Findings "Models", "STT backends",
"Live mic with whisper-stream", both "Push-to-talk" sections,
"large-v3-turbo", "The loop", "Overlay", "Barge-in", "Wake word trial",
"TTS" and "Kokoro-82M". The rest still describes the daemon: the tool,
context-size and tool-call-history findings are rules `assistant.py`
applies today.

Tools (2026-09-26): read-only shell-command TOML files in
`~/.config/desktop-assistant/tools/`; `config.toml` there overrides the
built-in settings. Examples in `examples/`. Spec:
`docs/superpowers/specs/2026-09-26-tools-and-config-design.md`.

Run: `python3 assistant.py` (`--test` for the self-check), or
`./assistant.sh restart` to replace a running one detached from the
terminal.

---

## Pipeline

```
socket (ask) ──> llama-server (LLM, tools) ──> socket (think, token, tool, done)
                                           └─> history.db (SQLite, FTS5)
```

- **Concurrency:** one reader thread per client, one reply thread per
  question. One lock guards the conversation, the turn counter and the
  database. `say()` checks the turn under it before each emit, and stop
  bumps the turn under it, so a stopped reply cannot send a token or tool
  event after its `done`. A late `ctx` can still arrive: `ask()` emits it
  outside the lock, and its numbers are true regardless.
- **One daemon:** `serve()` probe-connects first and refuses to start over a socket that answers; `main()` removes the socket only if it served it.
- **Commands** are type-checked before they run (ids must be `int`: SQLite would match `"1"` to row 1 while the Python comparison with the current id would not) and run under a try, so a failing command answers `error` instead of killing the client's reader.
- **Client sockets** have a 2 s timeout, not non-blocking mode: a large
  `hello` would not fit a non-blocking send and would drop the client.
- **Search** quotes the query as one FTS5 phrase with a prefix star. The
  self-check proves typed FTS syntax (`"`, `*`, `NOT`, `OR`, `)`) is
  matched as text and raises nothing.


## Hardware

- Discrete GPU: Intel Arc B580 (Battlemage). Target for both Vulkan and
  OpenVINO GPU.
- Integrated GPU: AMD Radeon 610M on a Ryzen 7 9700X, also exposed through
  Vulkan (RADV). Vulkan enumeration order is Arc first, iGPU second, llvmpipe
  last. Check device indices in the startup log before picking `-dev N`,
  the iGPU is a plausible wrong pick.

## whisper.cpp: what the package gives you

Built from ggml-org upstream 1.9.4 (SBo still ships 1.6.2, which is why it
is ours). Binaries only, libraries static: bundled ggml would clash with
llama.cpp-vulkan's.

Installed tools: `whisper-cli`, `whisper-server`, `whisper-bench`,
`whisper-quantize`, `whisper-vad-speech-segments`, `parakeet-cli`,
`parakeet-quantize`, and the SDL2 mic tools `whisper-stream` (live
transcription), `whisper-command` (voice commands), `whisper-talk-llama`,
`whisper-lsp`.

Backends compiled in: CPU, BLAS (OpenBLAS, autodetected), Vulkan
(`GGML_VULKAN`), ggml OpenVINO (`GGML_OPENVINO`, whole model through
OpenVINO), plus the older `WHISPER_OPENVINO` encoder-only path.

- Vulkan and ggml-openvino both register as GPU devices. whisper uses the
  first GPU it finds; choose with `whisper-cli -dev N`.
- `whisper-bench` has **no** device flag. Compare backends with
  `whisper-cli -dev N` on the same sample and read the timings it prints at
  exit.
- The `WHISPER_OPENVINO` encoder path needs the encoder converted to
  OpenVINO IR first (upstream README, "OpenVINO Support"). Not done yet.
  The ggml-openvino device does not need that conversion.
- OpenVINO libs are found through an rpath
  (`/usr/share/openvino/runtime/lib/intel64`); no need to source
  `setupvars.sh` at runtime. The OpenVINO GPU plugin
  (`libopenvino_intel_gpu_plugin.so`) is installed.
- Input goes through ffmpeg, so any format ffmpeg reads works.
- Speech encoders expect 16 kHz mono.

`whisper-talk-llama` already does mic, whisper, LLM in one binary, but it
loads its own model rather than talking to the running `llama-server`. Worth
reading as reference, probably not the final shape.

## llama-server: already running

A system service, `/etc/rc.d/rc.llama.cpp`, runs `llama-server` in **router
mode** on `http://localhost:8181` with an OpenAI-compatible API.

- Models are defined in `/etc/llama-server/presets.ini` (sections with
  `model =` and optional `mmproj =`), GGUF files under `/data/LLM-models/`.
- `--models-max 1`: the router keeps **one** model loaded. If the assistant
  asks for a different model than the one in use, the router swaps it out,
  under whoever else is using it (e.g. llamachat). Prefer reusing whatever
  model is loaded, or accept the reload latency knowingly.
- `/v1/models` lists models and each one's `architecture.input_modalities`.
  A model reporting `"audio"` accepts `input_audio` content parts directly,
  which is a possible shortcut that skips whisper entirely. Not evaluated.

Prior art: `../local-chat` (llamachat, PySide6) is a client for the same
router. It already records 16 kHz mono WAV and sends it as `input_audio`
to audio-capable models, and parses `presets.ini`. Reuse its findings, do
not import from it.

---

## Findings

### Attachments, live check (2026-10-16)

- An image reading "BLUE 42" (`magick`, 400x120) was answered `BLUE 42`,
  first token at 6.1 s, ctx 1242 / 32768.
- A two-page PDF, a text page and an image-only page, was answered with
  facts from both: the text page went inline, the image-only page as a
  `pdftoppm` render.
- A page render needs a bound: at `-r 150` a 3000 pt page came out
  6250x6250 at 171 MB peak (review probe). Renders use 150 dpi lowered to fit 1600 px; `-scale-to` also upscales, and a 600x300 pt scan sent as 2048x1024 with a second image took the model process down (router: `proxy error: Failed to read connection`).
- With the render fixed, the same scan (now 1250x625) plus the BLUE 42
  image, sent together, were answered from both in one reply, first token
  at 8.4 s, ctx 1615 / 32768; the model stayed loaded.
- **An image larger than llama-server's ubatch kills the model process.**
  Gemma 4's vision encoder is non-causal, and llama.cpp asserts
  `n_ubatch >= n_tokens` per image (`llama-context.cpp:1784`, in
  `/var/log/llama-server/server.log`). Tokens are 48 px patches: an A4 scan
  rendered at 1132x1601 became 1152x1584, 24x33 = 792 tokens, past the
  default ubatch of 512; 1250x625 (338) had worked, 2048x1024 (903) had
  not. The fix is the preset, not the client, since a large photo crosses
  512 as easily: `ubatch-size = 2048` and `batch-size = 2048` in
  `/etc/llama-server/presets.ini`, `[Gemma4-12B-qat-mtp]` (set 2026-10-16;
  check `/v1/models` args for `--ubatch-size 2048`). After it, the same
  two-page scan was answered, first token at 17.7 s, ctx 2782 / 32768.
- tesseract is installed with no language data, so OCR is the model's
  vision, not tesseract.
- A Unix socket path is limited to 108 bytes: a test socket under a deep
  scratch directory fails with `AF_UNIX path too long`.

### Text daemon, live check (2026-10-16)

Model Gemma4-12B-qat-mtp, ctx 32768, preset reasoning-budget 4096.

- Thinking on: `think` events arrive (1017 deltas for the symlink question;
  the first after 2.8 s). Reasoning took about 21 s, so the first token came
  at 23.8 s for a two-line question. This is why `thinking` became a config
  key (default true).
- "What time is it?": no tool call; the date stamp in the prompt answered
  it. First token 4.9 s.
- "How many unread mails do I have?": tool round in order `tool` mail
  done=false, `state` tool, `tool` mail done=true (0.12 s later), `state`
  thinking, tokens; first token 6.7 s.
- ctx used went 1166, 1247, 1275, 1603 across the three questions and the
  tool round.
- `list` returned the one conversation, `search` "symlink" returned it with
  a `**symlink**` snippet.
- History dir 700; `history.db`, `-wal` and `-shm` 600.
- SIGTERM logged `exit: SystemExit 0` and removed the socket.

### Models (2026-09-25)

Everything under `/data/voice-models/` (outside the repo, kept apart from
`/data/LLM-models/`):

- `whisper/ggml-base.bin`, 142 MB, multilingual. SHA1 matches upstream
  (`465707469ff3a37a2b9b8d8f89f2f99de7299dac`). Deleted 2026-09-26.
- `piper/it_IT-paola-medium.onnx` and `piper/en_US-lessac-medium.onnx`,
  61 MB each, plus their `.onnx.json`. Both output 22050 Hz. Deleted
  2026-09-26.
- `samples/it.wav`, `samples/en.wav`: 10 s, 16 kHz mono mic recordings used
  for the STT comparison. Personal voice data, never copy into the repo.

### STT backends (2026-09-25)

Same 10 s clip, `ggml-base`, `whisper-cli`:

| `-dev` | Device | Total (warm) | Result |
|--------|--------|--------------|--------|
| `-ng` | CPU, 4 threads | ~370-420 ms | correct |
| 0 | Vulkan0, Arc B580 | ~160-240 ms | correct |
| 1 | Vulkan1, iGPU | ~900 ms | correct |
| 2 | OPENVINO0 | 13.7 s | garbage |

**Decision: Arc on Vulkan, `-dev 0`, `-l auto`.**

- `-dev N` indexes GPU-type devices only, not the full device list:
  0 = Arc, 1 = iGPU, 2 = OpenVINO. An index past the last GPU silently falls
  back to CPU.
- The Arc's first run after a while costs 0.6-1.4 s (Vulkan pipeline
  warm-up), later runs are fast. A long-running process pays it once.
- `-l auto` detected both languages correctly (it, p=0.999) for ~40 ms extra
  (one more encoder pass). One setup covers Italian and English.
- ggml-openvino runs on **CPU**, not the Arc: the log says "OpenVINO: using
  device CPU" even with `GGML_OPENVINO_DEVICE=GPU`, and its output is
  garbage. Only OpenCL platform installed is Mesa `rusticl`; Intel
  compute-runtime (NEO), which the OpenVINO GPU plugin needs, is not
  installed. Likely cause, not confirmed. Decision on keeping OpenVINO in
  the package is deferred to `my-slackbuilds`.
- Every run tries to load `ggml-base-encoder-openvino.xml` (the
  `WHISPER_OPENVINO` encoder path), fails, and continues. Harmless noise.
- base accuracy is fine on full sentences, weak on short words: "ho" came
  out as "a", a spoken "hi" came out as "5". Try `ggml-small` (~488 MB) if
  that hurts short commands.

### Live mic with whisper-stream (2026-09-25)

`whisper-stream -m ggml-base.bin -l auto --step 0 --length 8000 -vth 0.6`
(VAD mode), 40 s, mixed Italian/English utterances with pauses.

- No `-dev` flag; it takes the first GPU, which is the Arc (Vulkan0). Default
  SDL capture device is the USB mic, correct.
- **Not usable as the assistant's STT entry point.** Its "VAD" is a crude
  energy check: on speech activity it re-transcribes the whole last
  `--length` window, not a segmented utterance. The same sentence comes out
  2-3 times, windows straddle two utterances in different languages (one
  output mixed an English sentence with the next Italian one), and `-l auto`
  detects per window, so mixed windows garble. A window with short words
  ("hello, hi, ciao") came out as Greek gibberish.
- Content of a clean window was mostly right ("ricordami di comprare
  latte"), so the model is fine, the segmentation is the problem.
- Conclusion: the loop must segment utterances itself (push-to-talk or a real
  VAD) and send exactly one utterance to whisper, kept warm in a
  long-running process (`whisper-server`) to avoid the Arc warm-up per call.

### Push-to-talk via the mic's mute button (2026-09-25)

The USB mic's hardware mute button toggles the ALSA control
`numid=2 'Mic Capture Switch'` on card 0 (`amixer -c 0 cget numid=2` reads
on/off, observed flipping with each press). So the button *is* the
push-to-talk: switch on = start recording, switch off = stop and send the
utterance to whisper. No keybind, no VAD.

- Event-driven, no polling: `amixer -c 0 events` prints a line on each
  control change ("Ready to listen..." then events).
- Card index 0 and numid 2 are what this machine shows now; card numbering
  can change with USB plug order, so resolve by card name, not number.
- The mic also exposes a HID keyboard device (`/dev/input/event3`, group
  `input`); not needed, the ALSA switch is enough.
- **Conflict: calls.** In a call the same button unmutes the mic for the
  call, and the assistant would record and answer the call. Decision: the
  assistant is paused while **presentation mode** is on (the user turns it on
  for calls). Presentation mode lives in the desktop status registry
  `~/bin/statusctl` (quickshell repo): `statusctl presentation get` prints
  `1`/`0`, backed by `$XDG_RUNTIME_DIR/status.presentation` (missing = off).
  Check it at the moment the switch goes on, and ignore the press if `1`.
  No watcher needed. Considered and not chosen: skipping when another app
  already records from the mic (`pactl list source-outputs`, sound server is
  PipeWire with the Pulse API); fallback if presentation mode gets forgotten.
- **Games (2026-09-26):** the shell's game detector writes a separate
  `status.gaming`, and the registry's effective presentation is
  `presentation OR gaming`. `statusctl presentation get` reads only the
  manual half: it printed `0` with pcsx2 running, so the assistant would
  have answered during games. `presenting()` now reads both files (the
  registry's documented interface). Live: press ignored with pcsx2
  running, normal again after it closed.

### Push-to-talk round trip with whisper-server (2026-09-25)

`whisper-server -m ggml-base.bin -dev 0 -l auto --host 127.0.0.1 --port 8182`
(port next to llama-server's 8181), throwaway bash loop: `amixer events`
drives `arecord` on/off, clip POSTed to `/inference`.

- Works end to end. Warm server: **0.10-0.23 s** per utterance, including the
  very first request (no per-call Arc warm-up once the server is up).
- `amixer events` piped into a loop is block-buffered and delivers nothing;
  needs `stdbuf -oL amixer ...`.
- Do **not** mute the mic from software (`amixer cset numid=2 off`): tried
  at loop startup and the user found it unreliable (likely out of sync with
  the button's own state/LED). The user drives mute/unmute on the button
  only; the loop just reads the current state at start.
- `amixer -c Microphone` (the card id) works, no need for the index.
- Stopping `arecord` with SIGINT leaves the WAV header unfinalized (size
  field reads as ~67108 s). whisper decodes it anyway; do not trust the
  header for the duration.
- **Language auto-detect is the weak spot with base on live speech:** an
  English sentence came out as a Greek/Hebrew/Arabic mix, "ciao" came out as
  "Tchau" (Portuguese). Italian sentence fine, "hi" fine. The same sentences
  from the pre-recorded samples detected correctly, so short or accented
  utterances push base's detector off. Only it/en will ever be used.
- Correction: the pre-recorded English sample *also* detects as Greek
  (el p=0.88, en 0.009, it 0.075), on server and CLI, CPU and GPU alike; the
  earlier CLI run decoded English text in spite of the wrong language tag.
  base's detector does not handle this speaker's English.
- Tried and rejected for base: (1) pick it/en by `language_probabilities`
  (verbose_json): picks it for English. (2) transcribe forced `it` and forced
  `en`, keep higher token-weighted `avg_logprob`: forced-it turns English into
  plausible Italian and scores *higher* (-0.164 vs -0.287). Moving to
  `ggml-small`.
- `ggml-small.bin` (466 MB, SHA1 `55356645c2b361a969dfd0ef2c5a50d530afd8d5`,
  matches upstream) in `/data/voice-models/whisper/` (deleted 2026-09-26). On the Arc via
  whisper-server: **~0.35 s per pass** on a 10 s clip. Transcripts clean on
  both samples (Italian and English self-introductions, all words right).
- small's detector *still* mislabels the English sample (la 0.45, it 0.31,
  en 0.04) though it decodes English text under auto. Not trustworthy.
- **Picker works with small:** forced `it` vs forced `en`, keep higher
  token-weighted `avg_logprob`. it.wav: it -0.196 vs en -0.426; en.wav:
  en -0.127 vs it -0.243. Both correct, ~0.5 s for the two passes.
  **STT decision: ggml-small + it/en picker.**
- Live push-to-talk with small + picker, 5 utterances: Italian sentence,
  English sentence, "hi", and a mixed it/en sentence all transcribed
  perfectly, 0.42-0.54 s each (two passes). "ciao" failed: forced `en`
  produced Chinese characters ("早, 早!") with the *better* logprob (-0.032
  vs -0.312); forced `it` gave "Ciao!". An initial `prompt` ("Ciao. Hi.")
  does not fix forced-en.
- **Fix: reject any candidate containing non-Latin letters** (it and en are
  both Latin script; `unicodedata.name(c).startswith("LATIN")` for every
  alphabetic char). With that guard all 5 live clips come out right.
- Mixed-language sentences: the picker chooses one language, but small still
  transcribes both halves correctly under either forced language.
- In the full loop the picker still failed: spoken English "What does HTML
  stand for?" -> picked it ("Che stanno per HTML?", it -0.153 vs en -0.214)
  though small's own detector said en 0.926 on that clip. Combining
  `logprob + w*log(p_detect)` over 8 saved clips: 7/8 at every w, never 8/8.
  small's two signals are each unreliable on this speaker, in different clips.

### large-v3-turbo (2026-09-25, **chosen**, replaces small)

`ggml-large-v3-turbo-q5_0.bin`, 548 MB, sha256 matches huggingface
(`394221709cd5ad1f40c46e6031ca61bce88931e6e088c188294c6d5a55ffa7e2`).

- Auto-detect correct on **all 7** single-language clips, including the
  English ones small's detector called Latin/Greek; the mixed it/en clip
  comes out "it". Transcripts good (spoken "HTML stands for" kept English).
- Its forced-language logprob picker is *worse* than small's (it.wav picks en,
  -0.225 vs -0.667): with turbo, trust the detector, drop the picker.
- Cost on the Arc: ~0.95 s (whisper-cli) / ~1.2 s (whisper-server) per
  clip. Encoder runs twice (~400 ms each): once for language detection, once
  for transcription.
- `-ac` (audio context) does not pay: 768 saves ~0.25 s but starts
  repeating words, 512 repeats more, 384 is garbage. Keep 0 (full).

### The loop, `assistant.py` (2026-09-25)

- STT: one `language=auto` pass on turbo; the answer is whichever of it/en
  the detector rates higher. Forced second pass only if detection lands on
  a third language or the text is non-Latin. whisper-server's default
  language is **en**, not auto: always send `language=auto` explicitly.
- Live: turbo still tagged one English question "it" ("What is
  slackware?"), but the text stayed English, so no harm, because:
- **TTS voice follows the reply's language, not the user's.** The model may
  answer in the other language (seen: user tagged en, reply Italian, read
  by the English voice). Per-sentence stopword count picks it/en.
- **Pad 0.5 s of silence** after each clip: whisper dropped the final word
  ("... a cosa serve [python]") when the button was pressed mid-word.
  Fixed by the padding. Recording is raw PCM (`arecord -t raw`), wrapped
  into a WAV with stdlib `wave`, which also fixes the SIGINT header problem.
- LLM: model = first one in `/v1/models` whose status is not `unloaded`
  (resident is Gemma4-12B, status `sleeping` when idle). Warm first token
  ~0.3 s; after `sleep-idle-seconds 1800` the first request waits ~45 s for
  the reload. `chat_template_kwargs.enable_thinking=false` sent; only
  `delta.content` is read.
- Timings live: STT ~1.23 s, first LLM sentence 0.6-0.8 s.
- Child processes (whisper-server, `amixer events`) get
  `prctl(PR_SET_PDEATHSIG, SIGTERM)` via ctypes in `preexec_fn`: they die
  with the assistant even on SIGKILL or a kill during startup (tested).
  Python's default SIGTERM skips `finally`; a handler calls `sys.exit`.
- 2026-09-28: `arecord` lacked it. The assistant died with no log line
  (last: "ready, mic is on", cause unknown) and left `arecord` recording
  into the unlinked raw file for ~76 min. It now gets `die_with_parent`
  too (not yet tested live); `assistant.sh status` lists a stray one.
- Gemma confidently gave a wrong fact (a city-to-city distance off by
  ~10x). No tools yet, so no time/weather/facts lookup.
- Recordings live in `$XDG_RUNTIME_DIR` and are deleted on exit.
- The reply's first word was often clipped: each reply opens a new `aplay`
  stream and the idle sink needs a moment to wake. 0.3 s of leading silence
  per reply fixed it (confirmed by ear).
- 2026-09-28: first word clipped again with the default sink on HDMI,
  which wakes slower than 0.3 s. Fixed outside the repo: a WirePlumber
  rule (`~/.config/wireplumber/wireplumber.conf.d/51-hdmi-no-suspend.conf`,
  `session.suspend-timeout-seconds = 0` for that node) keeps the sink
  awake; it now sits `IDLE`, not `SUSPENDED`, between replies. First
  word back (confirmed by ear).

### Tools, live test (2026-09-26)

Gemma4-12B through llama-server `--jinja`, streamed OpenAI `tool_calls`.
Eight spoken questions (time, calendar, home weather, remote forecast,
distance, a person, next appointment twice).

- Tool choice and arguments were right every time. Weather for a small
  town came with the country code the parameter description asks for.
- Bare city names are ambiguous for OpenWeatherMap: "Rome" and "Roma"
  both resolve to Rome, US. The place parameter asks for `City,CC` and
  the output names the country.
- **The model has no clock.** Asked for the next appointment it called the
  calendar with a 2025 start date, then recovered by calling `now` (three
  rounds, 4.0 s). With the date and time in the system prompt every turn:
  one round, 2.0 s.
- **Stale time from history** (2026-09-26). "Che ore sono?" asked 50
  minutes after the same question got the old time back, no tool call:
  the old answer in history won over the system prompt's clock. Measured
  on the live model, 6 runs each (time right / account filter set on a
  "spent from Mediolanum this month?" after three unrelated turns):
  clock in the system prompt only 4/6, 6/6; plus the same stamp
  `[Saturday 26 September 2026, 13:27]` on the current question only 6/6,
  6/6 (**chosen**); every history message stamped 6/6, **0/6**, it then
  called finance without `account` and said nothing matched. The stamp
  is not echoed in answers.
- **Gemma garbles numbers it spells out** ("tredici e trent settler",
  "trentusetzen", "duemiladistante"): 3 of 10 time answers at the preset's
  temperature 1.0, with or without the stamp. Temperature 0.6 gave 0/10
  but kept spelling. The default system prompt now asks for digits:
  20/20 "Sono le 13:37.", finance filter still 5/6. Kokoro's espeak
  reads Italian digits right (13:37, 26 settembre 2026, 6549,67 euro).
  English decimals come out split ("six thousand ... forty nine. sixty
  seven"): the phonemizer takes the point as a full stop.
- **Misheard words send it to the wrong tool.** Whisper wrote "Che ori
  sono?" (twice in one day); once the model answered with Rome's weather,
  "Rome,IT" being the example in the weather tool's place description.
  10 runs of "Che ori sono?": temperature 1.0 4 stray tool calls, 0.6
  1, 0.6 plus "the user's words come from speech recognition and may
  contain misheard words: read them by how they sound" in the system
  prompt 0 (2 of 10 gave the date only). Same hint: finance and weather
  questions still 10/10 on the right tool. "Che ore sono?" heard right:
  20/20 at either temperature.
- **Beam search in whisper** (2026-09-26). 10 clips of the user's short
  questions (it/en, 3 s, `/data/voice-models/samples/stt/`, personal voice
  data, never in the repo), `language=auto`: greedy 8/10 (0.80 s), beam 5
  9/10 (0.86 s), initial prompt of typical questions 9/10 but "Ho nuove
  mail?" came back as "." (lost question), beam + prompt 9/10. **Beam 5
  chosen**, prompt rejected. "ori" did not reproduce in these clips, so
  the misheard-word hint above stays.
- **It gets weekdays wrong from dates** ("mercoledi 29" for a Tuesday).
  `agenda.py` prints the weekday itself; right since.
- It searches for general knowledge too (a question about a well-known
  person), because the search description says "anything you are not
  sure about". Costs ~3 s (first sentence 6.7 s). **Kept on purpose** by
  the user: facts over speed.
- First sentence with one tool: 1.9-2.2 s for local tools, ~4 s with a
  search, 6.7 s for a search whose results the model read at length.
- Announcements heard on every slow tool, no audio glitch while the
  stream waits for the tool. Straight into the answer it sounded rushed;
  a 0.6 s pause after it fixed that (by ear).
- `agenda.py` matched khal's listing over 7 and 60 days on the real vdir,
  runs in ~60 ms.

### More tools, live test (2026-09-26)

`calc` (qalc), `mail` (mailctl search), `playing` (playerctl/mpc),
`finance` (firefly list commands). Every command is fixed in its tool
file: the model fills parameters but cannot reach `mailctl tag` or
`firefly tx add`.

- qalc reads "17.5% of 240" as a modulo. With the hint "percent as a
  factor: 240 * 17.5%" the model sent exactly that and got 42.
- "Days until Christmas": the model built `days(<today>; 2026-12-25)`
  from the date in the system prompt, right in one call (1.9 s).
- mail said "47 unread emails" for 47 *threads* (58 messages). The tool
  now prints "N messages in M threads".
- finance "this month" used the 30-day default and the model called it
  the month's total. Spending now takes a `since` date, empty = first of
  the month.
- "Spent on account X" had no account filter: the model fetched balances
  plus total spending and honestly said it could not split them. Added
  `account`; firefly's per-account listing needs `--until` or the API
  answers 422. An unknown name gets the user's asset accounts back (not
  firefly's own error, which lists every expense account); tool errors
  are now capped at 2000 characters like normal output.
- The model passed a lowercase account name; matching is
  case-insensitive, firefly accepted it.
- First sentence 1.9-3.6 s with local tools, ~5-6.6 s with firefly
  (two API calls when filtering by account).
- STT misheard "mail" as "snail"; the model answered about a snail,
  harmless for read-only tools. Phase 2 (modifying tools) needs
  confirmation before acting for exactly this reason.
- These tools speak personal data (mail subjects, amounts) out loud.

### Bluetooth and VM tools, live test (2026-09-26)

`bluetooth` (bluetoothctl show/devices/info) and `vms` (virsh list,
domstate, dominfo, domifaddr). Connecting devices and handling VMs are
phase 2.

- A game controller (DualSense) reports no `Battery Percentage` through
  BlueZ; its kernel driver (hid-playstation, same for hid-sony and
  hid-nintendo) exposes a power supply named after the device address.
  `bluetooth.sh` reads `/sys/class/power_supply/*<mac>*/capacity` (and
  `status` for charging) as a fallback. Live: "85 percento".
- `virsh domifaddr` with the default lease source is empty for a bridged
  guest; `--source arp` (then `agent`) has the address.
- **The model invented a fact about the user's system.** After listing
  the VMs, "what is buildsystem's IP?" got a made-up private address with
  no tool call: the previous tool output had no IP and the model filled
  the gap. Fix in the default system prompt: facts about the user's
  computer, devices, accounts, mail, calendar or money come only from a
  tool; if none can tell, say so, never guess. Same sequence after the
  fix: it called `vms(name)` and read the real address. A custom
  `system` in config.toml drops this rule unless it is copied in.
- Whisper wrote "build system" as two words; the model mapped it to the
  VM name `buildsystem` without help.

### Overlay (2026-09-26)

The assistant owns `$XDG_RUNTIME_DIR/desktop-assistant.sock` (0600) and
broadcasts JSON lines: `hello`, `state`, `ctx`, `level`; the quickshell
component `assistant/` draws them. Spec and plan under `docs/superpowers/`.

- Live event sequence for a search question: listening, transcribing,
  thinking, ctx, tool (search), thinking, ctx, speaking, idle. Barge-in
  shows as listening mid-reply; the overlay stays up.
- **arecord writes its file in periods**, ~125 ms by default: the user's
  levels came at 8 per second. `--period-time=33333` gives ~31 (measured
  as file-size changes over 2 s: 16 vs 63).
- Reply levels are emitted on a playback clock (lead silence + samples
  written / rate), not at synthesis; measured 30 per second of speech.
- `speaking` comes from the reply thread, not the speaker: otherwise the
  tool announcement replaced the `tool` state while the tool still ran.
- `ctx_max` from `GET /props?model=<id>` (`n_ctx`), read at startup too,
  else the first `hello` said 0.
- The overlay's ripple first ran on a 16 ms `Timer` and stuttered (by
  eye); `FrameAnimation` (vsync, Qt 6.11) plus a GPU canvas fixed it.
  Bars were replaced by three sine ripples at the user's request, 30%
  larger (`zoom`), and pulsing dots while transcribing or thinking.
- Ignored presses: `notify-send -u normal -p`, then `-r <id>` so repeated
  presses make one entry; DND holds it in the drawer (confirmed).
- Started from Hyprland autostart (hypr-theme repo), not an rc script:
  audio, the session bus and the runtime dir only exist in the login
  session. Log in `$XDG_RUNTIME_DIR/desktop-assistant.log` (transcripts,
  so private and gone at reboot).

### Context size (2026-09-26)

Gemma's context is 32768 tokens. Measured against the running model:
system prompt + question 106 prompt tokens, + the 10 tool definitions
946, + a full 20-message history of typical exchanges 1576. Worst case
per question is ~9k (20 long history messages ~3k, up to 3 tool rounds
of 2000-character results ~4k), so the 20-message window and the
2000-character tool cap keep it well inside; each extra tool costs ~85.

- The assistant logs `context: N prompt tokens` for every LLM round
  (`stream_options.include_usage`). Live: a weather question 969, then
  1115 with the tool result.
- The server log (`/var/log/llama-server/server.log`) also prints
  `stop processing: n_tokens = N, truncated = 0` per request, but it is
  shared with every router client, so it cannot be attributed to the
  assistant. `truncated = 1` there would mean an overflow.
- 2026-09-28: history now keeps tool calls with placeholder results (see
  "Tool calls in history"), window 10 whole turns. Each past call adds
  ~50 tokens, so the worst case goes from ~9k to ~10k (10 turns of 3
  calls). Full results kept would
  have been ~22k.

### Tool calls in history (2026-09-28)

"Ci sono aggiornamenti da installare?" after four exchanges (time, mail,
calc, calendar) got "12 updates, 3 security fixes, a kernel bump" with
no tool call: invented, the system was up to date. History kept only
what was said, so the model saw factual answers with no tool behind them
and copied that. Replayed against the live model, 10 runs each: that
history 1/10 called `updates`, empty history 10/10; a system prompt
line saying earlier answers came from hidden tool calls 0/10 (worse);
the same history with its tool calls kept 10/10, whether results were
kept whole, cut to 200 characters, or replaced by a placeholder "(shown
to the user at the time)" (**chosen**, cheapest). With the placeholder, a
follow-up on an old result ("chi ha mandato la terza email?") calls the
tool again, 10/10, no invented answer; costs one extra tool call. The
"never guess" rule alone does not hold once history shows tool-less
answers.

### Barge-in (2026-09-26)

A button press cuts the current reply. A global `turn` counter goes up on
every press; queued audio items carry their turn and the speaker drops
stale ones; the press kills the running `aplay` at once (that also drops
the ~1 s already in its pipe). `reply()` runs on its own thread and
returns when its turn is stale: after STT, between streamed pieces,
before tool rounds. The button loop is never blocked by a reply any more.

- Live: stopped instantly while speaking and while a search was running,
  nothing of the old reply played afterwards (by ear). A correction
  ("anzi, dimmi...") pressed during a search got its own clean answer.
- A reply whose text had already finished generating is cut in audio
  only; the log shows no `interrupted` then, and the exchange stays in
  history. A reply cut while generating is dropped from history whole,
  so user and assistant messages still alternate.
- Known ceilings (`ponytail:` comments): a running tool is not killed,
  its result is dropped; a press during the old reply's STT can
  overwrite the WAV mid-upload.
- Unrelated but seen: whisper garbled "i Mille" into "i 1000.000e" and
  the model invented an answer for it instead of searching.

### Slackware updates tool (2026-09-26)

`updates` compares the -current ChangeLog (NFS mirror) with
`/var/lib/pkgtools/packages`: the newest batch holding an entry whose exact
package name is installed is the last one applied, everything newer is
pending. Parses the 2.1 MB ChangeLog in ~95 ms, so no announcement.

- Removed entries are left out of that check: a removed package is still
  installed under its exact name and would mark the batch as applied.
- A package bumped twice keeps the security flag of the older bump.
- The first version listed every package (6.8k characters, 506 upgrades)
  and the 2000-character cap cut it. The user does not want lists read
  aloud: the summary gives counts, security fixes, the kernel bump and the
  short new/removed lists, and tells the model to offer a written recap.
  `report=true` writes `$XDG_RUNTIME_DIR/slackware-updates.md` and opens it
  in `MD_EDITOR` (typora in the user's config), detached.
- Live: counts, security fixes and "kernel 7.2.7" spoken, then it offered
  the recap on its own; "si" opened it (2.2 s).
- The overlay did not show after `restart.sh` (now `assistant.sh`)
  started it together with the assistant: quickshell's `Socket` never retries a failed connect. Fixed in
  the quickshell repo (fresh `Socket` per retry), which also covers login,
  where the overlay starts first.

### Wake word trial, openWakeWord (2026-09-28, undecided)

Logging-only trial of a wake word instead of the button, nothing wired
into the assistant. openWakeWord 0.6.0 in a throwaway venv
(`--system-site-packages`, `pip install --no-deps`), pretrained
`hey_jarvis_v0.1` on the ONNX backend. Models in
`/data/voice-models/openwakeword/` (3.6 MB, from the openWakeWord GitHub
release v0.5.1): `melspectrogram.onnx`, `embedding_model.onnx`,
`hey_jarvis_v0.1.onnx`. The logger read `arecord` 16 kHz mono in 80 ms
chunks, threshold 0.5, and logged hits, near misses (0.2-0.5) and the
mic switch state; the assistant was stopped meanwhile.

- The package imports scikit-learn at init only for custom verifier
  models; stubbing the module in `sys.modules` is enough.
  `tflite-runtime` has no Python 3.12 wheels; ONNX needs neither.
- The mic's mute switch gates the detector too: with it off ALSA
  delivers silence, so a wake word means an open mic, and the button can
  no longer be push-to-talk.
- Quiet room, 2 min take: 8/8 hits, peaks 0.66-0.999, no false wakes.
  Speech at -30 to -40 dBFS, noise floor -62 dBFS. The accent is fine.
- Music at low volume (floor -54.5 dBFS): 6/6 live, 5/5 in a second
  run, but peaks dropped to 0.54-0.96. The same utterance scored 0.465
  offline and 0.919 live: the score depends on where the 80 ms chunks
  fall, so near the threshold it is noisy.
- Music at the user's usual volume (floor -38 dBFS, voice only ~5 dB
  above it): **3/8**. Offline the misses peaked at 0.40-0.48; live two
  of them stayed under 0.2. Earlier, casual use with music gave 2 of 10+.
- No false wakes from music at either volume, nor from other background
  sound (~-50 dBFS): nothing but the user's attempts scored above 0.2.
- The score flickers under 0.2 for single frames mid-word: one utterance
  needs ~0.5 s under the floor before it counts as ended.
- **No echo cancellation available:** Slackware's pipewire 1.6.9 is built
  without `webrtc-audio-processing`; only `libspa-aec-null.so` is
  installed, and `pactl load-module module-echo-cancel` fails with "No
  such entity".
- Options left open by the user: a lower threshold (0.3 would have
  caught 6/8 live under loud music, false-wake rate at 0.3 unmeasured),
  maybe only while a player is playing; ducking the music on a near
  miss; packaging `webrtc-audio-processing` and rebuilding pipewire with
  it (a core package, decision for `my-slackbuilds`; would also cover
  the assistant's own voice); or keeping push-to-talk and the wake word
  only for quiet rooms.

### TTS (2026-09-25)

`piper-tts` on CPU (onnxruntime), ~5 s sentence per voice:

- CLI to WAV: 0.64-0.76 s wall for ~5 s of audio, ~7x real time.
- CLI `--output-raw`, time to first byte: ~0.58 s, nearly all voice load
  (0.46 s). Python + `import piper` alone is 0.09 s.
- Voice preloaded in-process (`PiperVoice.load` once, then
  `voice.synthesize(text)` yields chunks per sentence): first chunk in
  0.035-0.069 s.
- So: a per-reply `piper-tts` process costs ~0.5 s of load each time; a
  long-running process holding the voice makes TTS latency negligible.
- `--output-raw` is 16-bit mono PCM at the voice's rate (22050 Hz):
  `piper-tts -m <voice> --output-raw | aplay -r 22050 -f S16_LE -c 1`.
- **Quality rejected by ear** on a ~20 s paragraph: Italian `paola-medium`
  flat with odd rhythm, English `lessac-medium` metallic and robotic.
  Piper speed is fine, its voices are not. Looking at alternatives.

### Kokoro-82M (2026-09-25, **chosen**)

`kokoro-onnx` 0.6.1 in a throwaway venv (`--system-site-packages`, so it
uses the system onnxruntime 1.30, CPU). Model `kokoro-v1.0.onnx` (311 MB)
and `voices-v1.0.bin` (27 MB) in `/data/voice-models/kokoro/`, from the
kokoro-onnx GitHub release `model-files-v1.0`.

- The PyPI `espeakng-loader` wheel ships a lib with a hardcoded CI data path
  and fails ("Error processing file .../phontab"). Point it at the system
  espeak-ng instead:
  `EspeakConfig(lib_path="/usr/lib64/libespeak-ng.so.1", data_path="/usr/share/espeak-ng-data")`.
  A system package would do the same and drop `espeakng-loader`.
  Done 2026-09-26: `python3-kokoro-onnx` (with `python3-phonemizer`,
  `python3-dlinfo`) finds system espeak-ng with the default config.
- Load 0.30 s. `create()` on the ~20 s paragraph: ~2.6 s, ~7x real time,
  same for `if_sara`, `im_nicola` (lang `it`), `af_heart` (`en-us`),
  `bm_george` (`en-gb`). `create()` returns only after the whole text, so
  a reply needs per-sentence synthesis (or `create_stream`) to start
  speaking early.
- **By ear: much better than piper. Kokoro chosen.** Voices: `if_sara`
  (Italian) and `af_heart` (English), both female for consistency.
  `bm_george` also liked (male, "Jarvis"-like) but not used. `if_sara` wants
  a bit more color/expressiveness.
- Kokoro's only knob is `speed`; color comes from blending voice style
  vectors (`k.get_voice_style(name)` returns an array, `create(voice=...)`
  accepts it). Compared by ear: sara alone, sara 80/heart 20, sara 65/heart
  35, sara 80/bella 20. **Italian voice chosen: `0.8*if_sara + 0.2*af_bella`,
  lang `it`.**

---

## Next steps

- **Canvas:** the quickshell side, `desktop/assistant/`, its own plan in
  that repo.
- **Local memory:** `docs/superpowers/plans/2026-09-30-local-memory.md`
  still applies; its tasks 1-4 target this daemon, the voice-specific
  bits of task 4 aside.
- **llamachat ports**, if the canvas replaces llamachat for good:
  attachments, skills, a model picker, model-written titles.
- **Phase 2 tools:** tools that write or modify (events, messages,
  desktop), with a review step, own spec.
- **Voice:** start from `voice-v1`. The open items from that era (STT
  latency, wake word, OpenVINO) are in the sections above.

## Package-side leftovers (tracked in my-slackbuilds, not here)

- whisper.cpp not yet built on the buildsystem VM via slackrepo (the
  installed package is from the local build).
- `sbopkglint` on the openvino package not done.
- OpenCL-CLHPP 2026.05.29 (needed by openvino and ggml-openvino) exists
  only as a buildsystem hint; the Docker test-build needs a temporary local
  copy when its cache is cold. Permanent fix undecided.
- whisper.cpp and openvino cannot build on Slackware 15.0 (CMake 3.26,
  glslc, old CLHPP). -current only.

---

## Agent rules

- Ask before acting when anything is ambiguous, and before committing.
- Package changes belong in `my-slackbuilds`, not here.
- Never download models or voices without saying which and how big.
- Everything stays local: no cloud STT/TTS/LLM fallbacks.