# Voice input: design

Date: 2026-10-19. Status: implemented (2026-10-19), live-checked.

## Goal

Let the user speak a question to the canvas instead of typing it, with
no STT server: the resident model hears the clip itself. Only when the
loaded model reports `audio` in its `input_modalities`; whisper stays
out of the text daemon. Replies stay text (no TTS here).

## Verified before designing

Gemma4-12B-qat-mtp, 10 short clips with known text
(`/data/voice-models/samples/stt/`) plus two 10 s clips; AGENTS.md,
Findings, "Audio input".

- Asked for a transcript only, naming Italian or English: 10/10 right in
  meaning, ~0.55 s per 3 s clip. large-v3-turbo was 9/10 at 0.86 s.
- Asked to answer from the audio directly, with the real system prompt
  and tools: tool choice 13/14, but "Che ore sono?" was answered in
  Estonian or Spanish 5 times in 6. The same questions as text: 20/20.

So the daemon transcribes first and then asks the transcript as text.

## Protocol

`ask` gains an optional `audio`: an absolute path to a WAV file. `text`
must then be empty; `files` may come with it.

```json
{"type": "ask", "text": "", "audio": "/run/user/1000/assistant-mic.wav"}
```

As with `files`, the daemon reads the path itself and never deletes or
moves it: the canvas owns the recording.

`state` gains `transcribing`, sent before the transcript request. After
it the reply runs as a typed `ask` with the transcript as `text`: the
`user` event, history, search and the date stamp all see the words, not
the audio.

`audio` that is not a string, or comes with text, answers `bad audio`
(no `turn`, like `bad files`). Otherwise errors answer with `error`
(with `turn`) and nothing more is sent:

- not an absolute path, missing, not a readable WAV (stdlib `wave`);
  the length comes from the file size, since a recording stopped by a
  signal leaves the header's length unset;
- longer than 60 s: "recording too long". 60 s is about 1500 audio
  tokens; past the 2048 ubatch an image crashed the model, and audio
  has not been tried there;
- a model whose `input_modalities` lacks `audio`: "<model> cannot hear
  audio";
- an empty transcript: "heard nothing".

## Transcription

One non-streamed-to-the-client request, thinking off, the configured
temperature (0.6 when tested), no tools, no history, no system prompt:

- a text part, "Transcribe this audio exactly, in the language spoken
  (Italian or English). Output only the transcript.";
- an `input_audio` part, `{"data": <base64>, "format": "wav"}`, no
  `data:` prefix (llamachat's finding).

It goes through `ask()` with the reply's turn. The turn is pending from
the `transcribing` state on, so a stop while transcribing shuts the
request and ends with `done` (stopped) and `idle`, like any reply. The
transcript is stripped and goes to `answer()` unchanged.

## Testing

`python3 assistant.py --test` gains, against the fake server:

- an `audio` ask: the fake server receives the transcript request with
  the `input_audio` part, then a second request whose user message is the
  transcript text; the `user` event carries the transcript;
- the "cannot hear audio" error with `/models` reporting `["text",
  "image"]`;
- each refusal: relative path, not a WAV, over 60 s, empty transcript,
  `audio` with non-empty `text`.

Live: the 10 clips through the socket, with the transcript and tool
choice compared to the typed question.

## Not included

The microphone itself: recording, the button and the level meter belong
to the canvas, specified in the quickshell repo. Spoken replies, a wake
word, and keeping the audio in history.