diff options
| -rw-r--r-- | AGENTS.md | 32 | ||||
| -rw-r--r-- | docs/superpowers/specs/2026-10-19-voice-input-design.md | 86 |
2 files changed, 118 insertions, 0 deletions
@@ -120,6 +120,36 @@ not import from it. ## Findings +### Audio input, live check (2026-10-19) + +Gemma4-12B-qat-mtp hearing the clip itself (`input_audio`, base64 WAV, +no `data:` prefix), no whisper. The 10 clips in +`/data/voice-models/samples/stt/` (3 s each, `expected.tsv`) plus +`it.wav`/`en.wav` (10 s). Spec: +`docs/superpowers/specs/2026-10-19-voice-input-design.md`. + +- **Transcript only** ("Transcribe this audio exactly, in the language + spoken (Italian or English). Output only the transcript.", thinking + off, temperature 0.6): 10/10 right in meaning ("Ho nuove email." for + "Ho nuove mail?", "Ciao." for "Ciao"); both 10 s clips word for word. + ~0.55 s per 3 s clip, 0.9 s for 10 s, 1.3 s for the first (cold). + large-v3-turbo with beam 5 was 9/10 at 0.86 s. +- Cost about 25 prompt tokens per second of audio: 110 for a 3 s clip + with the prompt, 285 for 10 s. +- **Transcript and answer in one call**, with a prompt that named no + language: 2 of 10 transcripts came out Spanish ("¿Qué hora es?"), and + answers were in Spanish or English for Italian questions. +- **Answering from the audio directly**, real system prompt, stamp and + 12 tools, first round only, 2 runs per clip: tool choice 13/14 (one + "What time is it?" answered from the stamp, right). But the 6 "Che ore + sono?" runs, answered without a tool, came back in Estonian ("Praegu on + kell 10:35.") 2 times, Spanish 3 times (once a non-answer, "Hay 24 + horas en un día."), Italian once: 1/6 right. The same questions typed: + 20/20. ~70 more prompt tokens per clip. **Decision: transcribe first, + then ask the transcript as text.** History and search need the words + anyway. +- Not tried: a clip long enough to cross the 2048 ubatch (~80 s). + ### Attachments, live check (2026-10-16) - An image reading "BLUE 42" (`magick`, 400x120) was answered `BLUE 42`, @@ -732,6 +762,8 @@ kokoro-onnx GitHub release `model-files-v1.0`. - **Local memory:** `docs/superpowers/plans/2026-09-30-local-memory.md` still applies; its tasks 1-4 target this daemon, the voice-specific bits of task 4 aside. +- **Voice input:** `docs/superpowers/specs/2026-10-19-voice-input-design.md`, + the model transcribes, no whisper; the mic is the canvas's part. - **llamachat ports**, if the canvas replaces llamachat for good: attachments, skills, a model picker, model-written titles. - **Phase 2 tools:** tools that write or modify (events, messages, diff --git a/docs/superpowers/specs/2026-10-19-voice-input-design.md b/docs/superpowers/specs/2026-10-19-voice-input-design.md new file mode 100644 index 0000000..860967a --- /dev/null +++ b/docs/superpowers/specs/2026-10-19-voice-input-design.md @@ -0,0 +1,86 @@ +# Voice input: design + +Date: 2026-10-19. Status: proposed. + +## Goal + +Let the user speak a question to the canvas instead of typing it, with +no STT server: the resident model hears the clip itself. Only when the +loaded model reports `audio` in its `input_modalities`; whisper stays +out of the text daemon. Replies stay text (no TTS here). + +## Verified before designing + +Gemma4-12B-qat-mtp, 10 short clips with known text +(`/data/voice-models/samples/stt/`) plus two 10 s clips; AGENTS.md, +Findings, "Audio input". + +- Asked for a transcript only, naming Italian or English: 10/10 right in + meaning, ~0.55 s per 3 s clip. large-v3-turbo was 9/10 at 0.86 s. +- Asked to answer from the audio directly, with the real system prompt + and tools: tool choice 13/14, but "Che ore sono?" was answered in + Estonian or Spanish 5 times in 6. The same questions as text: 20/20. + +So the daemon transcribes first and then asks the transcript as text. + +## Protocol + +`ask` gains an optional `audio`: an absolute path to a WAV file. `text` +must then be empty; `files` may come with it. + +```json +{"type": "ask", "text": "", "audio": "/run/user/1000/assistant-mic.wav"} +``` + +As with `files`, the daemon reads the path itself and never deletes or +moves it: the canvas owns the recording. + +`state` gains `transcribing`, sent before the transcript request. After +it the reply runs as a typed `ask` with the transcript as `text`: the +`user` event, history, search and the date stamp all see the words, not +the audio. + +Errors answer with `error` (with `turn`) and nothing more is sent: + +- not an absolute path, missing, not a readable WAV (stdlib `wave`); +- longer than 60 s: "recording too long". 60 s is about 1500 audio + tokens; past the 2048 ubatch an image crashed the model, and audio + has not been tried there; +- a model whose `input_modalities` lacks `audio`: "<model> cannot hear + audio"; +- an empty transcript: "heard nothing". + +## Transcription + +One non-streamed-to-the-client request, thinking off, temperature 0.6, +no tools, no history, no system prompt: + +- a text part, "Transcribe this audio exactly, in the language spoken + (Italian or English). Output only the transcript."; +- an `input_audio` part, `{"data": <base64>, "format": "wav"}`, no + `data:` prefix (llamachat's finding). + +It goes through `ask()` with the reply's turn, so a stop while +transcribing shuts the request like any other. The transcript is +stripped and goes to `answer()` unchanged. + +## Testing + +`python3 assistant.py --test` gains, against the fake server: + +- an `audio` ask: the fake server receives the transcript request with + the `input_audio` part, then a second request whose user message is the + transcript text; the `user` event carries the transcript; +- the "cannot hear audio" error with `/models` reporting `["text", + "image"]`; +- each refusal: relative path, not a WAV, over 60 s, empty transcript, + `audio` with non-empty `text`. + +Live: the 10 clips through the socket, with the transcript and tool +choice compared to the typed question. + +## Not included + +The microphone itself: recording, the button and the level meter belong +to the canvas, specified in the quickshell repo. Spoken replies, a wake +word, and keeping the audio in history. |