diff options
| author | Danilo M. <danix@danix.xyz> | 2026-09-26 13:56:01 +0200 |
|---|---|---|
| committer | Danilo M. <danix@danix.xyz> | 2026-09-26 13:56:01 +0200 |
| commit | 7c8e8d641854d5583965b3fa8ae5ebb02ce7085a (patch) | |
| tree | ad893a9b1a7b5189f986dc44b02b34bff87547da | |
| parent | e372786e5d3c34133ebe18d7df8e1f9dac8329ca (diff) | |
| download | desktop-assistant-7c8e8d641854d5583965b3fa8ae5ebb02ce7085a.tar.gz desktop-assistant-7c8e8d641854d5583965b3fa8ae5ebb02ce7085a.zip | |
Use beam search in whisper, tell the model about misheard words
Whisper wrote "Che ori sono?" twice in one day and the model once
answered with the weather. On 10 recorded short questions beam search
(beam_size=5) gets 9 right against 8 for greedy, for 0.06 s more; an
initial prompt also got 9 but lost one question entirely, so it is not
used.
The default system prompt now says the user's words come from speech
recognition and should be read by how they sound: on "Che ori sono?"
stray tool calls went from 1 in 10 to none, other tools unaffected.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| -rw-r--r-- | AGENTS.md | 16 | ||||
| -rw-r--r-- | assistant.py | 4 |
2 files changed, 20 insertions, 0 deletions
@@ -343,6 +343,22 @@ distance, a person, next appointment twice). reads Italian digits right (13:37, 26 settembre 2026, 6549,67 euro). English decimals come out split ("six thousand ... forty nine. sixty seven"): the phonemizer takes the point as a full stop. +- **Misheard words send it to the wrong tool.** Whisper wrote "Che ori + sono?" (twice in one day); once the model answered with Rome's weather, + "Rome,IT" being the example in the weather tool's place description. + 10 runs of "Che ori sono?": temperature 1.0 4 stray tool calls, 0.6 + 1, 0.6 plus "the user's words come from speech recognition and may + contain misheard words: read them by how they sound" in the system + prompt 0 (2 of 10 gave the date only). Same hint: finance and weather + questions still 10/10 on the right tool. "Che ore sono?" heard right: + 20/20 at either temperature. +- **Beam search in whisper** (2026-09-26). 10 clips of the user's short + questions (it/en, 3 s, `/data/voice-models/samples/stt/`, personal voice + data, never in the repo), `language=auto`: greedy 8/10 (0.80 s), beam 5 + 9/10 (0.86 s), initial prompt of typical questions 9/10 but "Ho nuove + mail?" came back as "." (lost question), beam + prompt 9/10. **Beam 5 + chosen**, prompt rejected. "ori" did not reproduce in these clips, so + the misheard-word hint above stays. - **It gets weekdays wrong from dates** ("mercoledi 29" for a Tuesday). `agenda.py` prints the weekday itself; right since. - It searches for general knowledge too (a question about a well-known diff --git a/assistant.py b/assistant.py index bf6049a..991dfa0 100644 --- a/assistant.py +++ b/assistant.py @@ -52,6 +52,8 @@ DEFAULTS = { "briefly in plain sentences: no markdown, no lists, no emoji. " "Reply in the language the user speaks, Italian or English. " "Write numbers, times and dates in digits, never spelled out. " + "The user's words come from speech recognition and may contain " + "misheard words: read them by how they sound. " "Facts about the user's computer, devices, accounts, mail, calendar " "or money come only from a tool: call one, and if none can tell, " "say you do not know. Never guess them."), @@ -261,7 +263,9 @@ def envelope(audio, sr): def transcribe(lang="auto"): # Always send language: whisper-server's own default is en, not auto. + # Beam search: 9/10 test clips right vs 8/10 greedy, +0.06 s. out = subprocess.run(["curl", "-s", "-F", f"file=@{WAV}", "-F", f"language={lang}", + "-F", "beam_size=5", "-F", "response_format=verbose_json", WHISPER + "/inference"], capture_output=True, text=True).stdout d = json.loads(out) |