From ea603369bd342ed67b802fb04febfde9632e91e2 Mon Sep 17 00:00:00 2001 From: "Danilo M." Date: Mon, 19 Oct 2026 10:46:39 +0200 Subject: docs: spec voice input, the model transcribes Gemma4 heard 10/10 short clips right as a transcript, but answered straight from the audio in Estonian or Spanish 5 times in 6. So the daemon transcribes first and asks the words as text. Co-Authored-By: Claude Opus 5.5 --- .../specs/2026-10-19-voice-input-design.md | 86 ++++++++++++++++++++++ 1 file changed, 86 insertions(+) create mode 100644 docs/superpowers/specs/2026-10-19-voice-input-design.md (limited to 'docs/superpowers/specs') diff --git a/docs/superpowers/specs/2026-10-19-voice-input-design.md b/docs/superpowers/specs/2026-10-19-voice-input-design.md new file mode 100644 index 0000000..860967a --- /dev/null +++ b/docs/superpowers/specs/2026-10-19-voice-input-design.md @@ -0,0 +1,86 @@ +# Voice input: design + +Date: 2026-10-19. Status: proposed. + +## Goal + +Let the user speak a question to the canvas instead of typing it, with +no STT server: the resident model hears the clip itself. Only when the +loaded model reports `audio` in its `input_modalities`; whisper stays +out of the text daemon. Replies stay text (no TTS here). + +## Verified before designing + +Gemma4-12B-qat-mtp, 10 short clips with known text +(`/data/voice-models/samples/stt/`) plus two 10 s clips; AGENTS.md, +Findings, "Audio input". + +- Asked for a transcript only, naming Italian or English: 10/10 right in + meaning, ~0.55 s per 3 s clip. large-v3-turbo was 9/10 at 0.86 s. +- Asked to answer from the audio directly, with the real system prompt + and tools: tool choice 13/14, but "Che ore sono?" was answered in + Estonian or Spanish 5 times in 6. The same questions as text: 20/20. + +So the daemon transcribes first and then asks the transcript as text. + +## Protocol + +`ask` gains an optional `audio`: an absolute path to a WAV file. `text` +must then be empty; `files` may come with it. + +```json +{"type": "ask", "text": "", "audio": "/run/user/1000/assistant-mic.wav"} +``` + +As with `files`, the daemon reads the path itself and never deletes or +moves it: the canvas owns the recording. + +`state` gains `transcribing`, sent before the transcript request. After +it the reply runs as a typed `ask` with the transcript as `text`: the +`user` event, history, search and the date stamp all see the words, not +the audio. + +Errors answer with `error` (with `turn`) and nothing more is sent: + +- not an absolute path, missing, not a readable WAV (stdlib `wave`); +- longer than 60 s: "recording too long". 60 s is about 1500 audio + tokens; past the 2048 ubatch an image crashed the model, and audio + has not been tried there; +- a model whose `input_modalities` lacks `audio`: " cannot hear + audio"; +- an empty transcript: "heard nothing". + +## Transcription + +One non-streamed-to-the-client request, thinking off, temperature 0.6, +no tools, no history, no system prompt: + +- a text part, "Transcribe this audio exactly, in the language spoken + (Italian or English). Output only the transcript."; +- an `input_audio` part, `{"data": , "format": "wav"}`, no + `data:` prefix (llamachat's finding). + +It goes through `ask()` with the reply's turn, so a stop while +transcribing shuts the request like any other. The transcript is +stripped and goes to `answer()` unchanged. + +## Testing + +`python3 assistant.py --test` gains, against the fake server: + +- an `audio` ask: the fake server receives the transcript request with + the `input_audio` part, then a second request whose user message is the + transcript text; the `user` event carries the transcript; +- the "cannot hear audio" error with `/models` reporting `["text", + "image"]`; +- each refusal: relative path, not a WAV, over 60 s, empty transcript, + `audio` with non-empty `text`. + +Live: the 10 clips through the socket, with the transcript and tool +choice compared to the typed question. + +## Not included + +The microphone itself: recording, the button and the level meter belong +to the canvas, specified in the quickshell repo. Spoken replies, a wake +word, and keeping the audio in history. -- cgit v1.2.3