diff options
| author | Danilo M. <danix@danix.xyz> | 2026-09-25 22:26:46 +0200 |
|---|---|---|
| committer | Danilo M. <danix@danix.xyz> | 2026-09-25 22:26:46 +0200 |
| commit | 5f6db83a6618d665081f2d4ab5a771125562a1f3 (patch) | |
| tree | ff785b7e2a754c2192176ad3de9e88f35d04aa5a | |
| download | desktop-assistant-5f6db83a6618d665081f2d4ab5a771125562a1f3.tar.gz desktop-assistant-5f6db83a6618d665081f2d4ab5a771125562a1f3.zip | |
The microphone's mute button is the push-to-talk: the ALSA Mic Capture
Switch starts and stops a recording. whisper-server (large-v3-turbo, Vulkan
on the Arc) transcribes it, the model already resident in llama-server
answers, and Kokoro speaks the reply sentence by sentence.
Only Italian and English are spoken, so detection is restricted to those
two; the voice follows the language of the reply, not of the question.
Presses are ignored while presentation mode is on, so calls keep the mic.
AGENTS.md records the backend, model and voice comparisons behind these
choices. Licensed GPLv2 only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| -rw-r--r-- | .gitignore | 2 | ||||
| -rw-r--r-- | AGENTS.md | 397 | ||||
| -rw-r--r-- | CLAUDE.md | 6 | ||||
| -rw-r--r-- | LICENSE | 338 | ||||
| -rw-r--r-- | README.md | 62 | ||||
| -rw-r--r-- | assistant.py | 281 |
6 files changed, 1086 insertions, 0 deletions
diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..a230a78 --- /dev/null +++ b/.gitignore @@ -0,0 +1,2 @@ +.venv/ +__pycache__/ diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..899fd86 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,397 @@ +# desktop-assistant + +A local desktop voice assistant: speak, get a spoken answer, nothing leaves +the machine. Everything runs on already-packaged local tools; this repo is +the glue between them. + +Status: **first working loop** in `assistant.py` (2026-09-25). Mic button +on/off is push-to-talk, whisper large-v3-turbo on the Arc transcribes, +the resident llama-server model answers, Kokoro speaks it. Button-off to +first spoken word ~2 s. Kokoro is not packaged yet: it runs from the +repo's gitignored `.venv`. See "Findings" and "Next steps". + +Run: `.venv/bin/python assistant.py` (`--test` for the self-check). The +venv is `python3 -m venv --system-site-packages .venv && .venv/bin/pip +install kokoro-onnx`; system-site-packages gives it the system onnxruntime. + +--- + +## Pipeline + +``` +mic button ──> arecord ──> whisper-server (STT) ──> llama-server (LLM) ──> Kokoro (TTS) ──> aplay +``` + +| Stage | Package (installed) | Binary | Notes | +|-------|---------------------|--------|-------| +| STT | `whisper.cpp` 1.9.4 | `whisper-server` (also `whisper-stream`, `whisper-cli`) | SDL2 mic capture, ffmpeg decoding, Vulkan + OpenVINO backends | +| LLM | `llama.cpp-vulkan` 0.5.0 | `llama-server` | already running as a system service, see below | +| TTS | none yet: `kokoro-onnx` from PyPI in `.venv` | | uses system `python3-onnxruntime` and `espeak-ng` | +| (TTS, unused) | `piper-tts` 1.7.0 | `piper-tts` | voices rejected by ear, see Findings | + +The three packages are in the `my-slackbuilds` repo +(`../slackbuilds-superrepo/my-slackbuilds/`). Package-level changes (build +options, backends, bumps) happen there, under that repo's `AGENTS.md`, not +here. `openvino` 2026.4.0 is also packaged there, solely to feed whisper.cpp. + +## Hardware + +- Discrete GPU: Intel Arc B580 (Battlemage). Target for both Vulkan and + OpenVINO GPU. +- Integrated GPU: AMD Radeon 610M on a Ryzen 7 9700X, also exposed through + Vulkan (RADV). Vulkan enumeration order is Arc first, iGPU second, llvmpipe + last. Check device indices in the startup log before picking `-dev N`, + the iGPU is a plausible wrong pick. + +## whisper.cpp: what the package gives you + +Built from ggml-org upstream 1.9.4 (SBo still ships 1.6.2, which is why it +is ours). Binaries only, libraries static: bundled ggml would clash with +llama.cpp-vulkan's. + +Installed tools: `whisper-cli`, `whisper-server`, `whisper-bench`, +`whisper-quantize`, `whisper-vad-speech-segments`, `parakeet-cli`, +`parakeet-quantize`, and the SDL2 mic tools `whisper-stream` (live +transcription), `whisper-command` (voice commands), `whisper-talk-llama`, +`whisper-lsp`. + +Backends compiled in: CPU, BLAS (OpenBLAS, autodetected), Vulkan +(`GGML_VULKAN`), ggml OpenVINO (`GGML_OPENVINO`, whole model through +OpenVINO), plus the older `WHISPER_OPENVINO` encoder-only path. + +- Vulkan and ggml-openvino both register as GPU devices. whisper uses the + first GPU it finds; choose with `whisper-cli -dev N`. +- `whisper-bench` has **no** device flag. Compare backends with + `whisper-cli -dev N` on the same sample and read the timings it prints at + exit. +- The `WHISPER_OPENVINO` encoder path needs the encoder converted to + OpenVINO IR first (upstream README, "OpenVINO Support"). Not done yet. + The ggml-openvino device does not need that conversion. +- OpenVINO libs are found through an rpath + (`/usr/share/openvino/runtime/lib/intel64`); no need to source + `setupvars.sh` at runtime. The OpenVINO GPU plugin + (`libopenvino_intel_gpu_plugin.so`) is installed. +- Input goes through ffmpeg, so any format ffmpeg reads works. +- Speech encoders expect 16 kHz mono. + +`whisper-talk-llama` already does mic, whisper, LLM in one binary, but it +loads its own model rather than talking to the running `llama-server`. Worth +reading as reference, probably not the final shape. + +## llama-server: already running + +A system service, `/etc/rc.d/rc.llama.cpp`, runs `llama-server` in **router +mode** on `http://localhost:8181` with an OpenAI-compatible API. + +- Models are defined in `/etc/llama-server/presets.ini` (sections with + `model =` and optional `mmproj =`), GGUF files under `/data/LLM-models/`. +- `--models-max 1`: the router keeps **one** model loaded. If the assistant + asks for a different model than the one in use, the router swaps it out, + under whoever else is using it (e.g. llamachat). Prefer reusing whatever + model is loaded, or accept the reload latency knowingly. +- `/v1/models` lists models and each one's `architecture.input_modalities`. + A model reporting `"audio"` accepts `input_audio` content parts directly, + which is a possible shortcut that skips whisper entirely. Not evaluated. + +Prior art: `../local-chat` (llamachat, PySide6) is a client for the same +router. It already records 16 kHz mono WAV and sends it as `input_audio` +to audio-capable models, and parses `presets.ini`. Reuse its findings, do +not import from it. + +## piper-tts + +Prebuilt upstream wheel (OHF-voice/piper1-gpl), console script `piper-tts`, +Python module `piper` under the system python3.12. Runtime deps +`python3-onnxruntime` and `python3-pathvalidate` are packaged separately. +Voices are ONNX models plus a `.onnx.json` config; none downloaded yet. + +--- + +## Findings + +### Models (2026-09-25) + +Everything under `/data/voice-models/` (outside the repo, kept apart from +`/data/LLM-models/`): + +- `whisper/ggml-base.bin`, 142 MB, multilingual. SHA1 matches upstream + (`465707469ff3a37a2b9b8d8f89f2f99de7299dac`). +- `piper/it_IT-paola-medium.onnx` and `piper/en_US-lessac-medium.onnx`, + 61 MB each, plus their `.onnx.json`. Both output 22050 Hz. +- `samples/it.wav`, `samples/en.wav`: 10 s, 16 kHz mono mic recordings used + for the STT comparison. Personal voice data, never copy into the repo. + +### STT backends (2026-09-25) + +Same 10 s clip, `ggml-base`, `whisper-cli`: + +| `-dev` | Device | Total (warm) | Result | +|--------|--------|--------------|--------| +| `-ng` | CPU, 4 threads | ~370-420 ms | correct | +| 0 | Vulkan0, Arc B580 | ~160-240 ms | correct | +| 1 | Vulkan1, iGPU | ~900 ms | correct | +| 2 | OPENVINO0 | 13.7 s | garbage | + +**Decision: Arc on Vulkan, `-dev 0`, `-l auto`.** + +- `-dev N` indexes GPU-type devices only, not the full device list: + 0 = Arc, 1 = iGPU, 2 = OpenVINO. An index past the last GPU silently falls + back to CPU. +- The Arc's first run after a while costs 0.6-1.4 s (Vulkan pipeline + warm-up), later runs are fast. A long-running process pays it once. +- `-l auto` detected both languages correctly (it, p=0.999) for ~40 ms extra + (one more encoder pass). One setup covers Italian and English. +- ggml-openvino runs on **CPU**, not the Arc: the log says "OpenVINO: using + device CPU" even with `GGML_OPENVINO_DEVICE=GPU`, and its output is + garbage. Only OpenCL platform installed is Mesa `rusticl`; Intel + compute-runtime (NEO), which the OpenVINO GPU plugin needs, is not + installed. Likely cause, not confirmed. Decision on keeping OpenVINO in + the package is deferred to `my-slackbuilds`. +- Every run tries to load `ggml-base-encoder-openvino.xml` (the + `WHISPER_OPENVINO` encoder path), fails, and continues. Harmless noise. +- base accuracy is fine on full sentences, weak on short words: "ho" came + out as "a", a spoken "hi" came out as "5". Try `ggml-small` (~488 MB) if + that hurts short commands. + +### Live mic with whisper-stream (2026-09-25) + +`whisper-stream -m ggml-base.bin -l auto --step 0 --length 8000 -vth 0.6` +(VAD mode), 40 s, mixed Italian/English utterances with pauses. + +- No `-dev` flag; it takes the first GPU, which is the Arc (Vulkan0). Default + SDL capture device is the USB mic, correct. +- **Not usable as the assistant's STT entry point.** Its "VAD" is a crude + energy check: on speech activity it re-transcribes the whole last + `--length` window, not a segmented utterance. The same sentence comes out + 2-3 times, windows straddle two utterances in different languages (one + output mixed an English sentence with the next Italian one), and `-l auto` + detects per window, so mixed windows garble. A window with short words + ("hello, hi, ciao") came out as Greek gibberish. +- Content of a clean window was mostly right ("ricordami di comprare + latte"), so the model is fine, the segmentation is the problem. +- Conclusion: the loop must segment utterances itself (push-to-talk or a real + VAD) and send exactly one utterance to whisper, kept warm in a + long-running process (`whisper-server`) to avoid the Arc warm-up per call. + +### Push-to-talk via the mic's mute button (2026-09-25) + +The USB mic's hardware mute button toggles the ALSA control +`numid=2 'Mic Capture Switch'` on card 0 (`amixer -c 0 cget numid=2` reads +on/off, observed flipping with each press). So the button *is* the +push-to-talk: switch on = start recording, switch off = stop and send the +utterance to whisper. No keybind, no VAD. + +- Event-driven, no polling: `amixer -c 0 events` prints a line on each + control change ("Ready to listen..." then events). +- Card index 0 and numid 2 are what this machine shows now; card numbering + can change with USB plug order, so resolve by card name, not number. +- The mic also exposes a HID keyboard device (`/dev/input/event3`, group + `input`); not needed, the ALSA switch is enough. +- **Conflict: calls.** In a call the same button unmutes the mic for the + call, and the assistant would record and answer the call. Decision: the + assistant is paused while **presentation mode** is on (the user turns it on + for calls). Presentation mode lives in the desktop status registry + `~/bin/statusctl` (quickshell repo): `statusctl presentation get` prints + `1`/`0`, backed by `$XDG_RUNTIME_DIR/status.presentation` (missing = off). + Check it at the moment the switch goes on, and ignore the press if `1`. + No watcher needed. Considered and not chosen: skipping when another app + already records from the mic (`pactl list source-outputs`, sound server is + PipeWire with the Pulse API); fallback if presentation mode gets forgotten. + +### Push-to-talk round trip with whisper-server (2026-09-25) + +`whisper-server -m ggml-base.bin -dev 0 -l auto --host 127.0.0.1 --port 8182` +(port next to llama-server's 8181), throwaway bash loop: `amixer events` +drives `arecord` on/off, clip POSTed to `/inference`. + +- Works end to end. Warm server: **0.10-0.23 s** per utterance, including the + very first request (no per-call Arc warm-up once the server is up). +- `amixer events` piped into a loop is block-buffered and delivers nothing; + needs `stdbuf -oL amixer ...`. +- Do **not** mute the mic from software (`amixer cset numid=2 off`): tried + at loop startup and the user found it unreliable (likely out of sync with + the button's own state/LED). The user drives mute/unmute on the button + only; the loop just reads the current state at start. +- `amixer -c Microphone` (the card id) works, no need for the index. +- Stopping `arecord` with SIGINT leaves the WAV header unfinalized (size + field reads as ~67108 s). whisper decodes it anyway; do not trust the + header for the duration. +- **Language auto-detect is the weak spot with base on live speech:** an + English sentence came out as a Greek/Hebrew/Arabic mix, "ciao" came out as + "Tchau" (Portuguese). Italian sentence fine, "hi" fine. The same sentences + from the pre-recorded samples detected correctly, so short or accented + utterances push base's detector off. Only it/en will ever be used. +- Correction: the pre-recorded English sample *also* detects as Greek + (el p=0.88, en 0.009, it 0.075), on server and CLI, CPU and GPU alike; the + earlier CLI run decoded English text in spite of the wrong language tag. + base's detector does not handle this speaker's English. +- Tried and rejected for base: (1) pick it/en by `language_probabilities` + (verbose_json): picks it for English. (2) transcribe forced `it` and forced + `en`, keep higher token-weighted `avg_logprob`: forced-it turns English into + plausible Italian and scores *higher* (-0.164 vs -0.287). Moving to + `ggml-small`. +- `ggml-small.bin` (466 MB, SHA1 `55356645c2b361a969dfd0ef2c5a50d530afd8d5`, + matches upstream) in `/data/voice-models/whisper/`. On the Arc via + whisper-server: **~0.35 s per pass** on a 10 s clip. Transcripts clean on + both samples (Italian and English self-introductions, all words right). +- small's detector *still* mislabels the English sample (la 0.45, it 0.31, + en 0.04) though it decodes English text under auto. Not trustworthy. +- **Picker works with small:** forced `it` vs forced `en`, keep higher + token-weighted `avg_logprob`. it.wav: it -0.196 vs en -0.426; en.wav: + en -0.127 vs it -0.243. Both correct, ~0.5 s for the two passes. + **STT decision: ggml-small + it/en picker.** +- Live push-to-talk with small + picker, 5 utterances: Italian sentence, + English sentence, "hi", and a mixed it/en sentence all transcribed + perfectly, 0.42-0.54 s each (two passes). "ciao" failed: forced `en` + produced Chinese characters ("早, 早!") with the *better* logprob (-0.032 + vs -0.312); forced `it` gave "Ciao!". An initial `prompt` ("Ciao. Hi.") + does not fix forced-en. +- **Fix: reject any candidate containing non-Latin letters** (it and en are + both Latin script; `unicodedata.name(c).startswith("LATIN")` for every + alphabetic char). With that guard all 5 live clips come out right. +- Mixed-language sentences: the picker chooses one language, but small still + transcribes both halves correctly under either forced language. +- In the full loop the picker still failed: spoken English "What does HTML + stand for?" -> picked it ("Che stanno per HTML?", it -0.153 vs en -0.214) + though small's own detector said en 0.926 on that clip. Combining + `logprob + w*log(p_detect)` over 8 saved clips: 7/8 at every w, never 8/8. + small's two signals are each unreliable on this speaker, in different clips. + +### large-v3-turbo (2026-09-25, **chosen**, replaces small) + +`ggml-large-v3-turbo-q5_0.bin`, 548 MB, sha256 matches huggingface +(`394221709cd5ad1f40c46e6031ca61bce88931e6e088c188294c6d5a55ffa7e2`). + +- Auto-detect correct on **all 7** single-language clips, including the + English ones small's detector called Latin/Greek; the mixed it/en clip + comes out "it". Transcripts good (spoken "HTML stands for" kept English). +- Its forced-language logprob picker is *worse* than small's (it.wav picks en, + -0.225 vs -0.667): with turbo, trust the detector, drop the picker. +- Cost on the Arc: ~0.95 s (whisper-cli) / ~1.2 s (whisper-server) per + clip. Encoder runs twice (~400 ms each): once for language detection, once + for transcription. +- `-ac` (audio context) does not pay: 768 saves ~0.25 s but starts + repeating words, 512 repeats more, 384 is garbage. Keep 0 (full). + +### The loop, `assistant.py` (2026-09-25) + +- STT: one `language=auto` pass on turbo; the answer is whichever of it/en + the detector rates higher. Forced second pass only if detection lands on + a third language or the text is non-Latin. whisper-server's default + language is **en**, not auto: always send `language=auto` explicitly. +- Live: turbo still tagged one English question "it" ("What is + slackware?"), but the text stayed English, so no harm, because: +- **TTS voice follows the reply's language, not the user's.** The model may + answer in the other language (seen: user tagged en, reply Italian, read + by the English voice). Per-sentence stopword count picks it/en. +- **Pad 0.5 s of silence** after each clip: whisper dropped the final word + ("... a cosa serve [python]") when the button was pressed mid-word. + Fixed by the padding. Recording is raw PCM (`arecord -t raw`), wrapped + into a WAV with stdlib `wave`, which also fixes the SIGINT header problem. +- LLM: model = first one in `/v1/models` whose status is not `unloaded` + (resident is Gemma4-12B, status `sleeping` when idle). Warm first token + ~0.3 s; after `sleep-idle-seconds 1800` the first request waits ~45 s for + the reload. `chat_template_kwargs.enable_thinking=false` sent; only + `delta.content` is read. +- Timings live: STT ~1.23 s, first LLM sentence 0.6-0.8 s. +- Child processes (whisper-server, `amixer events`) get + `prctl(PR_SET_PDEATHSIG, SIGTERM)` via ctypes in `preexec_fn`: they die + with the assistant even on SIGKILL or a kill during startup (tested). + Python's default SIGTERM skips `finally`; a handler calls `sys.exit`. +- Gemma confidently gave a wrong fact (a city-to-city distance off by + ~10x). No tools yet, so no time/weather/facts lookup. +- Recordings live in `$XDG_RUNTIME_DIR` and are deleted on exit. +- The reply's first word was often clipped: each reply opens a new `aplay` + stream and the idle sink needs a moment to wake. 0.3 s of leading silence + per reply fixed it (confirmed by ear). + +### TTS (2026-09-25) + +`piper-tts` on CPU (onnxruntime), ~5 s sentence per voice: + +- CLI to WAV: 0.64-0.76 s wall for ~5 s of audio, ~7x real time. +- CLI `--output-raw`, time to first byte: ~0.58 s, nearly all voice load + (0.46 s). Python + `import piper` alone is 0.09 s. +- Voice preloaded in-process (`PiperVoice.load` once, then + `voice.synthesize(text)` yields chunks per sentence): first chunk in + 0.035-0.069 s. +- So: a per-reply `piper-tts` process costs ~0.5 s of load each time; a + long-running process holding the voice makes TTS latency negligible. +- `--output-raw` is 16-bit mono PCM at the voice's rate (22050 Hz): + `piper-tts -m <voice> --output-raw | aplay -r 22050 -f S16_LE -c 1`. +- **Quality rejected by ear** on a ~20 s paragraph: Italian `paola-medium` + flat with odd rhythm, English `lessac-medium` metallic and robotic. + Piper speed is fine, its voices are not. Looking at alternatives. + +### Kokoro-82M (2026-09-25, **chosen**) + +`kokoro-onnx` 0.6.1 in a throwaway venv (`--system-site-packages`, so it +uses the system onnxruntime 1.30, CPU). Model `kokoro-v1.0.onnx` (311 MB) +and `voices-v1.0.bin` (27 MB) in `/data/voice-models/kokoro/`, from the +kokoro-onnx GitHub release `model-files-v1.0`. + +- The PyPI `espeakng-loader` wheel ships a lib with a hardcoded CI data path + and fails ("Error processing file .../phontab"). Point it at the system + espeak-ng instead: + `EspeakConfig(lib_path="/usr/lib64/libespeak-ng.so.1", data_path="/usr/share/espeak-ng-data")`. + A system package would do the same and drop `espeakng-loader`. +- Load 0.30 s. `create()` on the ~20 s paragraph: ~2.6 s, ~7x real time, + same for `if_sara`, `im_nicola` (lang `it`), `af_heart` (`en-us`), + `bm_george` (`en-gb`). `create()` returns only after the whole text, so + a reply needs per-sentence synthesis (or `create_stream`) to start + speaking early. +- **By ear: much better than piper. Kokoro chosen.** Voices: `if_sara` + (Italian) and `af_heart` (English), both female for consistency. + `bm_george` also liked (male, "Jarvis"-like) but not used. `if_sara` wants + a bit more color/expressiveness. +- Kokoro's only knob is `speed`; color comes from blending voice style + vectors (`k.get_voice_style(name)` returns an array, `create(voice=...)` + accepts it). Compared by ear: sara alone, sara 80/heart 20, sara 65/heart + 35, sara 80/bella 20. **Italian voice chosen: `0.8*if_sara + 0.2*af_bella`, + lang `it`.** + +--- + +## Next steps + +Steps 1-5 of the original plan are done (models, backend comparison, live +mic, TTS, glue); push-to-talk came for free from the mic button. Open, in +no particular order: + +- **Packaging Kokoro** in `my-slackbuilds` (`kokoro-onnx` without + `espeakng-loader`, pointed at system espeak-ng), then drop `.venv`. +- **Run as a service/autostart** (it currently runs by hand). +- **Barge-in:** pressing the button while it speaks does not stop the + reply yet. +- **Check presentation mode during games:** `statusctl presentation get` + reads the stored flag; confirm it is 1 while a game holds it on. +- **Latency:** STT ~1.2 s is the biggest share; the double encoder pass + (detect + transcribe) is the target if it matters. +- **Tools** (time, weather, calendar) so the model stops guessing facts. +- Piper leftovers: package unused, voices in `/data/voice-models/piper/` + (~120 MB) can go. ggml-base and ggml-small too, once turbo is settled. +- OpenVINO in the whisper.cpp package: runs on CPU only here (no Intel + compute-runtime), decision goes to `my-slackbuilds`. + +Open questions: whether an audio-capable LLM makes whisper unnecessary. +License: GPLv2 only (chosen 2026-09-25). + +## Package-side leftovers (tracked in my-slackbuilds, not here) + +- whisper.cpp not yet built on the buildsystem VM via slackrepo (the + installed package is from the local build). +- `sbopkglint` on the openvino package not done. +- OpenCL-CLHPP 2026.05.29 (needed by openvino and ggml-openvino) exists + only as a buildsystem hint; the Docker test-build needs a temporary local + copy when its cache is cold. Permanent fix undecided. +- whisper.cpp and openvino cannot build on Slackware 15.0 (CMake 3.26, + glslc, old CLHPP). -current only. + +--- + +## Agent rules + +- Ask before acting when anything is ambiguous, and before committing. +- Package changes belong in `my-slackbuilds`, not here. +- Never download models or voices without saying which and how big. +- Everything stays local: no cloud STT/TTS/LLM fallbacks. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..dfe9f3f --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,6 @@ +# desktop-assistant + +This file is intentionally thin. AGENTS.md is the single source of truth for +every agent tool. Edit AGENTS.md instead. + +@AGENTS.md @@ -0,0 +1,338 @@ + GNU GENERAL PUBLIC LICENSE + Version 2, June 1991 + + Copyright (C) 1989, 1991 Free Software Foundation, Inc., + <https://fsf.org/> + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The licenses for most software are designed to take away your +freedom to share and change it. By contrast, the GNU General Public +License is intended to guarantee your freedom to share and change free +software--to make sure the software is free for all its users. This +General Public License applies to most of the Free Software +Foundation's software and to any other program whose authors commit to +using it. (Some other Free Software Foundation software is covered by +the GNU Lesser General Public License instead.) You can apply it to +your programs, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +this service if you wish), that you receive source code or can get it +if you want it, that you can change the software or use pieces of it +in new free programs; and that you know you can do these things. + + To protect your rights, we need to make restrictions that forbid +anyone to deny you these rights or to ask you to surrender the rights. +These restrictions translate to certain responsibilities for you if you +distribute copies of the software, or if you modify it. + + For example, if you distribute copies of such a program, whether +gratis or for a fee, you must give the recipients all the rights that +you have. You must make sure that they, too, receive or can get the +source code. And you must show them these terms so they know their +rights. + + We protect your rights with two steps: (1) copyright the software, and +(2) offer you this license which gives you legal permission to copy, +distribute and/or modify the software. + + Also, for each author's protection and ours, we want to make certain +that everyone understands that there is no warranty for this free +software. If the software is modified by someone else and passed on, we +want its recipients to know that what they have is not the original, so +that any problems introduced by others will not reflect on the original +authors' reputations. + + Finally, any free program is threatened constantly by software +patents. We wish to avoid the danger that redistributors of a free +program will individually obtain patent licenses, in effect making the +program proprietary. To prevent this, we have made it clear that any +patent must be licensed for everyone's free use or not licensed at all. + + The precise terms and conditions for copying, distribution and +modification follow. + + GNU GENERAL PUBLIC LICENSE + TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION + + 0. This License applies to any program or other work which contains +a notice placed by the copyright holder saying it may be distributed +under the terms of this General Public License. The "Program", below, +refers to any such program or work, and a "work based on the Program" +means either the Program or any derivative work under copyright law: +that is to say, a work containing the Program or a portion of it, +either verbatim or with modifications and/or translated into another +language. (Hereinafter, translation is included without limitation in +the term "modification".) Each licensee is addressed as "you". + +Activities other than copying, distribution and modification are not +covered by this License; they are outside its scope. The act of +running the Program is not restricted, and the output from the Program +is covered only if its contents constitute a work based on the +Program (independent of having been made by running the Program). +Whether that is true depends on what the Program does. + + 1. You may copy and distribute verbatim copies of the Program's +source code as you receive it, in any medium, provided that you +conspicuously and appropriately publish on each copy an appropriate +copyright notice and disclaimer of warranty; keep intact all the +notices that refer to this License and to the absence of any warranty; +and give any other recipients of the Program a copy of this License +along with the Program. + +You may charge a fee for the physical act of transferring a copy, and +you may at your option offer warranty protection in exchange for a fee. + + 2. You may modify your copy or copies of the Program or any portion +of it, thus forming a work based on the Program, and copy and +distribute such modifications or work under the terms of Section 1 +above, provided that you also meet all of these conditions: + + a) You must cause the modified files to carry prominent notices + stating that you changed the files and the date of any change. + + b) You must cause any work that you distribute or publish, that in + whole or in part contains or is derived from the Program or any + part thereof, to be licensed as a whole at no charge to all third + parties under the terms of this License. + + c) If the modified program normally reads commands interactively + when run, you must cause it, when started running for such + interactive use in the most ordinary way, to print or display an + announcement including an appropriate copyright notice and a + notice that there is no warranty (or else, saying that you provide + a warranty) and that users may redistribute the program under + these conditions, and telling the user how to view a copy of this + License. (Exception: if the Program itself is interactive but + does not normally print such an announcement, your work based on + the Program is not required to print an announcement.) + +These requirements apply to the modified work as a whole. If +identifiable sections of that work are not derived from the Program, +and can be reasonably considered independent and separate works in +themselves, then this License, and its terms, do not apply to those +sections when you distribute them as separate works. But when you +distribute the same sections as part of a whole which is a work based +on the Program, the distribution of the whole must be on the terms of +this License, whose permissions for other licensees extend to the +entire whole, and thus to each and every part regardless of who wrote it. + +Thus, it is not the intent of this section to claim rights or contest +your rights to work written entirely by you; rather, the intent is to +exercise the right to control the distribution of derivative or +collective works based on the Program. + +In addition, mere aggregation of another work not based on the Program +with the Program (or with a work based on the Program) on a volume of +a storage or distribution medium does not bring the other work under +the scope of this License. + + 3. You may copy and distribute the Program (or a work based on it, +under Section 2) in object code or executable form under the terms of +Sections 1 and 2 above provided that you also do one of the following: + + a) Accompany it with the complete corresponding machine-readable + source code, which must be distributed under the terms of Sections + 1 and 2 above on a medium customarily used for software interchange; or, + + b) Accompany it with a written offer, valid for at least three + years, to give any third party, for a charge no more than your + cost of physically performing source distribution, a complete + machine-readable copy of the corresponding source code, to be + distributed under the terms of Sections 1 and 2 above on a medium + customarily used for software interchange; or, + + c) Accompany it with the information you received as to the offer + to distribute corresponding source code. (This alternative is + allowed only for noncommercial distribution and only if you + received the program in object code or executable form with such + an offer, in accord with Subsection b above.) + +The source code for a work means the preferred form of the work for +making modifications to it. For an executable work, complete source +code means all the source code for all modules it contains, plus any +associated interface definition files, plus the scripts used to +control compilation and installation of the executable. However, as a +special exception, the source code distributed need not include +anything that is normally distributed (in either source or binary +form) with the major components (compiler, kernel, and so on) of the +operating system on which the executable runs, unless that component +itself accompanies the executable. + +If distribution of executable or object code is made by offering +access to copy from a designated place, then offering equivalent +access to copy the source code from the same place counts as +distribution of the source code, even though third parties are not +compelled to copy the source along with the object code. + + 4. You may not copy, modify, sublicense, or distribute the Program +except as expressly provided under this License. Any attempt +otherwise to copy, modify, sublicense or distribute the Program is +void, and will automatically terminate your rights under this License. +However, parties who have received copies, or rights, from you under +this License will not have their licenses terminated so long as such +parties remain in full compliance. + + 5. You are not required to accept this License, since you have not +signed it. However, nothing else grants you permission to modify or +distribute the Program or its derivative works. These actions are +prohibited by law if you do not accept this License. Therefore, by +modifying or distributing the Program (or any work based on the +Program), you indicate your acceptance of this License to do so, and +all its terms and conditions for copying, distributing or modifying +the Program or works based on it. + + 6. Each time you redistribute the Program (or any work based on the +Program), the recipient automatically receives a license from the +original licensor to copy, distribute or modify the Program subject to +these terms and conditions. You may not impose any further +restrictions on the recipients' exercise of the rights granted herein. +You are not responsible for enforcing compliance by third parties to +this License. + + 7. If, as a consequence of a court judgment or allegation of patent +infringement or for any other reason (not limited to patent issues), +conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot +distribute so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you +may not distribute the Program at all. For example, if a patent +license would not permit royalty-free redistribution of the Program by +all those who receive copies directly or indirectly through you, then +the only way you could satisfy both it and this License would be to +refrain entirely from distribution of the Program. + +If any portion of this section is held invalid or unenforceable under +any particular circumstance, the balance of the section is intended to +apply and the section as a whole is intended to apply in other +circumstances. + +It is not the purpose of this section to induce you to infringe any +patents or other property right claims or to contest validity of any +such claims; this section has the sole purpose of protecting the +integrity of the free software distribution system, which is +implemented by public license practices. Many people have made +generous contributions to the wide range of software distributed +through that system in reliance on consistent application of that +system; it is up to the author/donor to decide if he or she is willing +to distribute software through any other system and a licensee cannot +impose that choice. + +This section is intended to make thoroughly clear what is believed to +be a consequence of the rest of this License. + + 8. If the distribution and/or use of the Program is restricted in +certain countries either by patents or by copyrighted interfaces, the +original copyright holder who places the Program under this License +may add an explicit geographical distribution limitation excluding +those countries, so that distribution is permitted only in or among +countries not thus excluded. In such case, this License incorporates +the limitation as if written in the body of this License. + + 9. The Free Software Foundation may publish revised and/or new versions +of the General Public License from time to time. Such new versions will +be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + +Each version is given a distinguishing version number. If the Program +specifies a version number of this License which applies to it and "any +later version", you have the option of following the terms and conditions +either of that version or of any later version published by the Free +Software Foundation. If the Program does not specify a version number of +this License, you may choose any version ever published by the Free Software +Foundation. + + 10. If you wish to incorporate parts of the Program into other free +programs whose distribution conditions are different, write to the author +to ask for permission. For software which is copyrighted by the Free +Software Foundation, write to the Free Software Foundation; we sometimes +make exceptions for this. Our decision will be guided by the two goals +of preserving the free status of all derivatives of our free software and +of promoting the sharing and reuse of software generally. + + NO WARRANTY + + 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY +FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN +OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES +PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED +OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS +TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE +PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, +REPAIR OR CORRECTION. + + 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR +REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, +INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING +OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED +TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY +YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER +PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE +POSSIBILITY OF SUCH DAMAGES. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +convey the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + <one line to give the program's name and a brief idea of what it does.> + Copyright (C) <year> <name of author> + + This program is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 2 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License along + with this program; if not, see <https://www.gnu.org/licenses/>. + +Also add information on how to contact you by electronic and paper mail. + +If the program is interactive, make it output a short notice like this +when it starts in an interactive mode: + + Gnomovision version 69, Copyright (C) year name of author + Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. + This is free software, and you are welcome to redistribute it + under certain conditions; type `show c' for details. + +The hypothetical commands `show w' and `show c' should show the appropriate +parts of the General Public License. Of course, the commands you use may +be called something other than `show w' and `show c'; they could even be +mouse-clicks or menu items--whatever suits your program. + +You should also get your employer (if you work as a programmer) or your +school, if any, to sign a "copyright disclaimer" for the program, if +necessary. Here is a sample; alter the names: + + Yoyodyne, Inc., hereby disclaims all copyright interest in the program + `Gnomovision' (which makes passes at compilers) written by James Hacker. + + <signature of Moe Ghoul>, 1 April 1989 + Moe Ghoul, President of Vice + +This General Public License does not permit incorporating your program into +proprietary programs. If your program is a subroutine library, you may +consider it more useful to permit linking proprietary applications with the +library. If this is what you want to do, use the GNU Lesser General +Public License instead of this License. diff --git a/README.md b/README.md new file mode 100644 index 0000000..d9f4d28 --- /dev/null +++ b/README.md @@ -0,0 +1,62 @@ +# desktop-assistant + +A local push-to-talk voice assistant. Unmute the microphone, speak, mute it: +the assistant transcribes what you said, asks a local LLM, and speaks the +answer. Italian and English. Nothing leaves the machine. + +``` +mic button ──> arecord ──> whisper-server (STT) ──> llama-server (LLM) ──> Kokoro (TTS) ──> aplay +``` + +## How it works + +- **Push-to-talk** is the microphone's own mute button: the script listens to + the ALSA `Mic Capture Switch` (`amixer events`). Switch on starts a + recording, switch off ends it. +- **Speech to text**: whisper.cpp's `whisper-server` with the + `large-v3-turbo` model on the GPU (Vulkan). The script starts it if it is + not already running and stops it on exit. +- **LLM**: an OpenAI-compatible `llama-server` on `localhost:8181`. The + script uses whichever model the server already has loaded, so it never + swaps models under other clients. +- **Text to speech**: Kokoro-82M through `kokoro-onnx`, one voice per + language, chosen per sentence from the language of the reply. Speech + starts after the first sentence, while the rest is still being generated. +- **Calls**: while the desktop's presentation mode is on + (`statusctl presentation get` returns `1`), button presses are ignored, so + a call can use the microphone. + +## Requirements + +- whisper.cpp (`whisper-server`) and a whisper ggml model +- a running `llama-server` with an OpenAI-compatible API +- Python 3 with `onnxruntime`, `numpy` and `kokoro-onnx` +- `espeak-ng`, ALSA utils (`amixer`, `arecord`, `aplay`), `curl` +- a USB microphone with a hardware mute button exposed as an ALSA switch + +Paths, the card name and ports are constants at the top of `assistant.py`. + +## Usage + +```sh +python3 -m venv --system-site-packages .venv +.venv/bin/pip install kokoro-onnx +.venv/bin/python assistant.py # run +.venv/bin/python assistant.py --test # self-check +``` + +## License + +Copyright (C) 2026 Danilo M. <danix@danix.xyz> + +This program is free software; you can redistribute it and/or modify it +under the terms of the GNU General Public License version 2 as published by +the Free Software Foundation. See [LICENSE](LICENSE). + +## Development Approach + +This project is developed using AI-assisted tools. Code is generated with the help of AI based on human-provided specifications, design decisions, and iterative feedback. + +All contributions are reviewed, tested, and curated by the maintainer before being included in the codebase. AI is used as a productivity and exploration tool, while human oversight remains central to all decisions. + +The goal is to combine the flexibility of AI-assisted development with standard open-source practices such as transparency, review, and accountability. diff --git a/assistant.py b/assistant.py new file mode 100644 index 0000000..49fb082 --- /dev/null +++ b/assistant.py @@ -0,0 +1,281 @@ +#!/usr/bin/env python3 +# desktop-assistant: push-to-talk local voice assistant. +# Copyright (C) 2026 Danilo M. <danix@danix.xyz> +# +# This program is free software; you can redistribute it and/or modify it +# under the terms of the GNU General Public License version 2 as published +# by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General +# Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +"""Mic button on: record. Mic button off: transcribe (whisper-server), +ask the loaded llama-server model, speak the reply (Kokoro). + +Run: .venv/bin/python assistant.py (--test runs the self-check) +""" + +import ctypes +import json +import os +import queue +import re +import signal +import subprocess +import sys +import threading +import time +import unicodedata +import urllib.request +import wave + +CARD = "Microphone" # ALSA card id of the USB mic +MODELS = "/data/voice-models" +WHISPER = "http://127.0.0.1:8182" +LLM = "http://localhost:8181/v1" +RAW = os.path.join(os.environ.get("XDG_RUNTIME_DIR", "/tmp"), "desktop-assistant.raw") +WAV = RAW[:-3] + "wav" +SYSTEM = ("You are a voice assistant. Your replies are spoken aloud, so answer " + "briefly in plain sentences: no markdown, no lists, no emoji. " + "Reply in the language the user speaks, Italian or English.") +KLANG = {"it": "it", "en": "en-us"} + + +def log(*a): + print(time.strftime("%H:%M:%S"), *a, flush=True) + + +def mic_on(): + out = subprocess.run(["amixer", "-c", CARD, "cget", "numid=2"], + capture_output=True, text=True).stdout + return out.rstrip().endswith("values=on") + + +def presenting(): + # During calls the user turns on presentation mode; the button then + # belongs to the call, not to us. + r = subprocess.run([os.path.expanduser("~/bin/statusctl"), "presentation", "get"], + capture_output=True, text=True) + return r.stdout.strip() == "1" + + +def latin(text): + return all(unicodedata.name(c, "").startswith("LATIN") for c in text if c.isalpha()) + + +def transcribe(lang="auto"): + # Always send language: whisper-server's own default is en, not auto. + out = subprocess.run(["curl", "-s", "-F", f"file=@{WAV}", "-F", f"language={lang}", + "-F", "response_format=verbose_json", WHISPER + "/inference"], + capture_output=True, text=True).stdout + d = json.loads(out) + return d.get("language_probabilities", {}), " ".join(d["text"].split()) + + +def stt(): + # large-v3-turbo's detector is reliable for this speaker (small's was + # not). Only it and en are spoken: if detection lands elsewhere, or the + # text is non-Latin (a hallucination, both are Latin script), redo the + # pass forced to the likelier of the two. + p, text = transcribe() + lang = max(KLANG, key=lambda l: p.get(l, 0)) + if max(p, key=p.get, default=lang) != lang or not latin(text): + text = transcribe(lang)[1] + return lang, text + + +IT_WORDS = set("il lo la gli le di che è non per una sono ho hai con mi ti ci del della " + "come questo questa anche ma più se nel alla puoi posso sei".split()) +EN_WORDS = set("the is are you i to of and it that for with not this what be have " + "can your my do there".split()) + + +def lang_of(sentence, fallback): + # The voice must follow the reply's language, not the user's: the model + # may answer in the other one. ponytail: stopword count, a real language + # identifier if this misfires. + words = re.findall(r"[a-zàèéìòù]+", sentence.lower()) + it = sum(w in IT_WORDS for w in words) + en = sum(w in EN_WORDS for w in words) + return "it" if it > en else "en" if en > it else fallback + + +def split_sentences(buf): + parts = re.split(r"(?<=[.!?…])\s+|\n+", buf) + return [p.strip() for p in parts[:-1] if p.strip()], parts[-1] + + +def llm_model(): + # The router keeps one model resident; asking for another would swap it + # out under other clients, so use whatever is not unloaded. + with urllib.request.urlopen(LLM + "/models") as r: + models = json.load(r)["data"] + return next((m["id"] for m in models if m["status"]["value"] != "unloaded"), + models[0]["id"]) + + +def ask(messages): + body = {"model": llm_model(), "messages": messages, "stream": True, + "chat_template_kwargs": {"enable_thinking": False}} + req = urllib.request.Request(LLM + "/chat/completions", json.dumps(body).encode(), + {"Content-Type": "application/json"}) + with urllib.request.urlopen(req) as r: + for line in r: + if not line.startswith(b"data: ") or line.strip() == b"data: [DONE]": + continue + choices = json.loads(line[6:])["choices"] + if choices and choices[0]["delta"].get("content"): + yield choices[0]["delta"]["content"] + + +def speaker(q): + from kokoro_onnx import Kokoro, EspeakConfig + # The espeakng-loader wheel's bundled lib has a broken data path; use + # the system espeak-ng. + k = Kokoro(f"{MODELS}/kokoro/kokoro-v1.0.onnx", f"{MODELS}/kokoro/voices-v1.0.bin", + espeak_config=EspeakConfig(lib_path="/usr/lib64/libespeak-ng.so.1", + data_path="/usr/share/espeak-ng-data")) + voices = {"it": 0.8 * k.get_voice_style("if_sara") + 0.2 * k.get_voice_style("af_bella"), + "en": k.get_voice_style("af_heart")} + play = None + while True: + item = q.get() + if item is None: # end of reply + if play: + play.stdin.close() + play.wait() + play = None + continue + text, lang = item + audio, sr = k.create(text, voice=voices[lang], lang=KLANG[lang]) + if play is None: + play = subprocess.Popen(["aplay", "-q", "-r", str(sr), "-f", "S16_LE", "-c", "1"], + stdin=subprocess.PIPE) + # A new stream wakes the suspended sink and the first ~0.2 s is + # lost: lead with silence so it eats that, not the first word. + play.stdin.write(bytes(int(sr * 0.3) * 2)) + # The pipe buffer lets synthesis of the next sentence overlap playback. + play.stdin.write((audio.clip(-1, 1) * 32767).astype("<i2").tobytes()) + + +def die_with_parent(): + # Runs in the child before exec: the kernel sends SIGTERM when we die, + # however we die, so no whisper-server or amixer is ever orphaned. + ctypes.CDLL("libc.so.6").prctl(1, signal.SIGTERM) # PR_SET_PDEATHSIG + + +def start_whisper(): + def up(): + try: + urllib.request.urlopen(WHISPER, timeout=1) + return True + except OSError: + return False + if up(): + return + p = subprocess.Popen(["whisper-server", "-m", f"{MODELS}/whisper/ggml-large-v3-turbo-q5_0.bin", + "-dev", "0", "--host", "127.0.0.1", "--port", WHISPER.rsplit(":", 1)[1]], + stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, + preexec_fn=die_with_parent) + for _ in range(60): + if up(): + return + time.sleep(0.5) + p.kill() + sys.exit("whisper-server did not come up") + + +def reply(history, q): + pcm = open(RAW, "rb").read() + if len(pcm) < 16000 * 2 * 0.3: # under 0.3 s: a stray press + return + # Trailing silence: whisper drops the last word when audio ends mid-word, + # which it does when the button is pressed while still talking. + with wave.open(WAV, "wb") as w: + w.setnchannels(1) + w.setsampwidth(2) + w.setframerate(16000) + w.writeframes(pcm + bytes(16000 * 2 // 2)) + t = time.perf_counter() + lang, text = stt() + if not text or text.startswith("["): # [BLANK_AUDIO] and friends + return + log(f"you ({lang}, {time.perf_counter() - t:.2f}s): {text}") + history.append({"role": "user", "content": text}) + t, said, buf = time.perf_counter(), [], "" + for delta in ask([{"role": "system", "content": SYSTEM}] + history[-20:]): + # ponytail: last 20 messages, summarise if long sessions need memory + buf += delta + done, buf = split_sentences(buf) + if not said and done: + log(f"first sentence after {time.perf_counter() - t:.2f}s") + for s in done: + said.append(s) + lang = lang_of(s, lang) + q.put((re.sub(r"[*_#`]", "", s), lang)) + if buf.strip(): + if not said: + log(f"first sentence after {time.perf_counter() - t:.2f}s") + said.append(buf.strip()) + q.put((re.sub(r"[*_#`]", "", buf.strip()), lang_of(buf, lang))) + q.put(None) + history.append({"role": "assistant", "content": " ".join(said)}) + log("assistant:", " ".join(said)) + + +def main(): + # SIGTERM must run the cleanup below too, or the recording is left behind. + signal.signal(signal.SIGTERM, lambda *_: sys.exit(0)) + start_whisper() + q = queue.Queue() + threading.Thread(target=speaker, args=(q,), daemon=True).start() + history, rec = [], None + events = subprocess.Popen(["stdbuf", "-oL", "amixer", "-c", CARD, "events"], + stdout=subprocess.PIPE, text=True, preexec_fn=die_with_parent) + prev = mic_on() + log(f"ready, mic is {'on' if prev else 'off'}") + try: + for line in events.stdout: + if "Mic Capture Switch" not in line: + continue + on = mic_on() + if on == prev: + continue + prev = on + if on and presenting(): + log("presentation mode: ignored") + elif on: + # ponytail: no barge-in, a new press does not cut the reply + rec = subprocess.Popen(["arecord", "-q", "-t", "raw", "-f", "S16_LE", "-r", "16000", + "-c", "1", RAW], stderr=subprocess.DEVNULL) + elif rec: + rec.send_signal(signal.SIGINT) + rec.wait() + rec = None + reply(history, q) + except KeyboardInterrupt: + pass + finally: + for f in (RAW, WAV): # the user's voice: never keep it + if os.path.exists(f): + os.remove(f) + + +def test(): + assert split_sentences("Ciao! Come stai? Io") == (["Ciao!", "Come stai?"], "Io") + assert split_sentences("Pi is 3.14 today") == ([], "Pi is 3.14 today") + assert split_sentences("one\n\ntwo") == (["one"], "two") + assert latin("Perché no? Hi!") and not latin("早, 早!") + assert lang_of("Non posso fornire il meteo in tempo reale.", "en") == "it" + assert lang_of("That is a very exciting place to be.", "it") == "en" + assert lang_of("Tokyo!", "it") == "it" + print("ok") + + +if __name__ == "__main__": + test() if sys.argv[1:] == ["--test"] else main() |
