1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
|
# Voice input: design
Date: 2026-10-19. Status: implemented (2026-10-19), live-checked.
## Goal
Let the user speak a question to the canvas instead of typing it, with
no STT server: the resident model hears the clip itself. Only when the
loaded model reports `audio` in its `input_modalities`; whisper stays
out of the text daemon. Replies stay text (no TTS here).
## Verified before designing
Gemma4-12B-qat-mtp, 10 short clips with known text
(`/data/voice-models/samples/stt/`) plus two 10 s clips; AGENTS.md,
Findings, "Audio input".
- Asked for a transcript only, naming Italian or English: 10/10 right in
meaning, ~0.55 s per 3 s clip. large-v3-turbo was 9/10 at 0.86 s.
- Asked to answer from the audio directly, with the real system prompt
and tools: tool choice 13/14, but "Che ore sono?" was answered in
Estonian or Spanish 5 times in 6. The same questions as text: 20/20.
So the daemon transcribes first and then asks the transcript as text.
## Protocol
`ask` gains an optional `audio`: an absolute path to a WAV file. `text`
must then be empty; `files` may come with it.
```json
{"type": "ask", "text": "", "audio": "/run/user/1000/assistant-mic.wav"}
```
As with `files`, the daemon reads the path itself and never deletes or
moves it: the canvas owns the recording.
`state` gains `transcribing`, sent before the transcript request. After
it the reply runs as a typed `ask` with the transcript as `text`: the
`user` event, history, search and the date stamp all see the words, not
the audio.
`audio` that is not a string, or comes with text, answers `bad audio`
(no `turn`, like `bad files`). Otherwise errors answer with `error`
(with `turn`) and nothing more is sent:
- not an absolute path, missing, not a readable WAV (stdlib `wave`);
the length comes from the file size, since a recording stopped by a
signal leaves the header's length unset;
- longer than 60 s: "recording too long". 60 s is about 1500 audio
tokens; past the 2048 ubatch an image crashed the model, and audio
has not been tried there;
- a model whose `input_modalities` lacks `audio`: "<model> cannot hear
audio";
- an empty transcript: "heard nothing".
## Transcription
One non-streamed-to-the-client request, thinking off, the configured
temperature (0.6 when tested), no tools, no history, no system prompt:
- a text part, "Transcribe this audio exactly, in the language spoken
(Italian or English). Output only the transcript.";
- an `input_audio` part, `{"data": <base64>, "format": "wav"}`, no
`data:` prefix (llamachat's finding).
It goes through `ask()` with the reply's turn. The turn is pending from
the `transcribing` state on, so a stop while transcribing shuts the
request and ends with `done` (stopped) and `idle`, like any reply. The
transcript is stripped and goes to `answer()` unchanged.
## Testing
`python3 assistant.py --test` gains, against the fake server:
- an `audio` ask: the fake server receives the transcript request with
the `input_audio` part, then a second request whose user message is the
transcript text; the `user` event carries the transcript;
- the "cannot hear audio" error with `/models` reporting `["text",
"image"]`;
- each refusal: relative path, not a WAV, over 60 s, empty transcript,
`audio` with non-empty `text`.
Live: the 10 clips through the socket, with the transcript and tool
choice compared to the typed question.
## Not included
The microphone itself: recording, the button and the level meter belong
to the canvas, specified in the quickshell repo. Spoken replies, a wake
word, and keeping the audio in history.
|
