From 7c8e8d641854d5583965b3fa8ae5ebb02ce7085a Mon Sep 17 00:00:00 2001 From: "Danilo M." Date: Sat, 26 Sep 2026 13:56:01 +0200 Subject: Use beam search in whisper, tell the model about misheard words Whisper wrote "Che ori sono?" twice in one day and the model once answered with the weather. On 10 recorded short questions beam search (beam_size=5) gets 9 right against 8 for greedy, for 0.06 s more; an initial prompt also got 9 but lost one question entirely, so it is not used. The default system prompt now says the user's words come from speech recognition and should be read by how they sound: on "Che ori sono?" stray tool calls went from 1 in 10 to none, other tools unaffected. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) (limited to 'AGENTS.md') diff --git a/AGENTS.md b/AGENTS.md index c04754c..10d7c3e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -343,6 +343,22 @@ distance, a person, next appointment twice). reads Italian digits right (13:37, 26 settembre 2026, 6549,67 euro). English decimals come out split ("six thousand ... forty nine. sixty seven"): the phonemizer takes the point as a full stop. +- **Misheard words send it to the wrong tool.** Whisper wrote "Che ori + sono?" (twice in one day); once the model answered with Rome's weather, + "Rome,IT" being the example in the weather tool's place description. + 10 runs of "Che ori sono?": temperature 1.0 4 stray tool calls, 0.6 + 1, 0.6 plus "the user's words come from speech recognition and may + contain misheard words: read them by how they sound" in the system + prompt 0 (2 of 10 gave the date only). Same hint: finance and weather + questions still 10/10 on the right tool. "Che ore sono?" heard right: + 20/20 at either temperature. +- **Beam search in whisper** (2026-09-26). 10 clips of the user's short + questions (it/en, 3 s, `/data/voice-models/samples/stt/`, personal voice + data, never in the repo), `language=auto`: greedy 8/10 (0.80 s), beam 5 + 9/10 (0.86 s), initial prompt of typical questions 9/10 but "Ho nuove + mail?" came back as "." (lost question), beam + prompt 9/10. **Beam 5 + chosen**, prompt rejected. "ori" did not reproduce in these clips, so + the misheard-word hint above stays. - **It gets weekdays wrong from dates** ("mercoledi 29" for a Tuesday). `agenda.py` prints the weekday itself; right since. - It searches for general knowledge too (a question about a well-known -- cgit v1.2.3