diff options
Diffstat (limited to 'AGENTS.md')
| -rw-r--r-- | AGENTS.md | 32 |
1 files changed, 32 insertions, 0 deletions
@@ -120,6 +120,36 @@ not import from it. ## Findings +### Audio input, live check (2026-10-19) + +Gemma4-12B-qat-mtp hearing the clip itself (`input_audio`, base64 WAV, +no `data:` prefix), no whisper. The 10 clips in +`/data/voice-models/samples/stt/` (3 s each, `expected.tsv`) plus +`it.wav`/`en.wav` (10 s). Spec: +`docs/superpowers/specs/2026-10-19-voice-input-design.md`. + +- **Transcript only** ("Transcribe this audio exactly, in the language + spoken (Italian or English). Output only the transcript.", thinking + off, temperature 0.6): 10/10 right in meaning ("Ho nuove email." for + "Ho nuove mail?", "Ciao." for "Ciao"); both 10 s clips word for word. + ~0.55 s per 3 s clip, 0.9 s for 10 s, 1.3 s for the first (cold). + large-v3-turbo with beam 5 was 9/10 at 0.86 s. +- Cost about 25 prompt tokens per second of audio: 110 for a 3 s clip + with the prompt, 285 for 10 s. +- **Transcript and answer in one call**, with a prompt that named no + language: 2 of 10 transcripts came out Spanish ("¿Qué hora es?"), and + answers were in Spanish or English for Italian questions. +- **Answering from the audio directly**, real system prompt, stamp and + 12 tools, first round only, 2 runs per clip: tool choice 13/14 (one + "What time is it?" answered from the stamp, right). But the 6 "Che ore + sono?" runs, answered without a tool, came back in Estonian ("Praegu on + kell 10:35.") 2 times, Spanish 3 times (once a non-answer, "Hay 24 + horas en un día."), Italian once: 1/6 right. The same questions typed: + 20/20. ~70 more prompt tokens per clip. **Decision: transcribe first, + then ask the transcript as text.** History and search need the words + anyway. +- Not tried: a clip long enough to cross the 2048 ubatch (~80 s). + ### Attachments, live check (2026-10-16) - An image reading "BLUE 42" (`magick`, 400x120) was answered `BLUE 42`, @@ -732,6 +762,8 @@ kokoro-onnx GitHub release `model-files-v1.0`. - **Local memory:** `docs/superpowers/plans/2026-09-30-local-memory.md` still applies; its tasks 1-4 target this daemon, the voice-specific bits of task 4 aside. +- **Voice input:** `docs/superpowers/specs/2026-10-19-voice-input-design.md`, + the model transcribes, no whisper; the mic is the canvas's part. - **llamachat ports**, if the canvas replaces llamachat for good: attachments, skills, a model picker, model-written titles. - **Phase 2 tools:** tools that write or modify (events, messages, |