diff options
| -rw-r--r-- | AGENTS.md | 39 | ||||
| -rw-r--r-- | README.md | 41 |
2 files changed, 77 insertions, 3 deletions
@@ -9,6 +9,13 @@ on/off is push-to-talk, whisper large-v3-turbo on the Arc transcribes, the resident llama-server model answers, Kokoro speaks it. Button-off to first spoken word ~2 s. See "Findings" and "Next steps". +Tools (2026-09-26, phase 1 of 2): the model can call read-only tools +(time, calendar, weather, search) defined as shell-command TOML files in +`~/.config/desktop-assistant/tools/`; `config.toml` there overrides the +built-in settings. Examples in `examples/`. STT, LLM and TTS stay local; +only the weather and search tools reach the network, and only when a +question needs them. Spec: `docs/superpowers/specs/2026-09-26-tools-and-config-design.md`. + Run: `python3 assistant.py` (`--test` for the self-check). --- @@ -295,6 +302,35 @@ drives `arecord` on/off, clip POSTed to `/inference`. stream and the idle sink needs a moment to wake. 0.3 s of leading silence per reply fixed it (confirmed by ear). +### Tools, live test (2026-09-26) + +Gemma4-12B through llama-server `--jinja`, streamed OpenAI `tool_calls`. +Eight spoken questions (time, calendar, home weather, remote forecast, +distance, a person, next appointment twice). + +- Tool choice and arguments were right every time. Weather for a small + town came with the country code the parameter description asks for. +- Bare city names are ambiguous for OpenWeatherMap: "Rome" and "Roma" + both resolve to Rome, US. The place parameter asks for `City,CC` and + the output names the country. +- **The model has no clock.** Asked for the next appointment it called the + calendar with a 2025 start date, then recovered by calling `now` (three + rounds, 4.0 s). With the date and time in the system prompt every turn: + one round, 2.0 s. +- **It gets weekdays wrong from dates** ("mercoledi 29" for a Tuesday). + `agenda.py` prints the weekday itself; right since. +- It searches for general knowledge too (a question about a well-known + person), because the search description says "anything you are not + sure about". Costs ~3 s (first sentence 6.7 s). **Kept on purpose** by + the user: facts over speed. +- First sentence with one tool: 1.9-2.2 s for local tools, ~4 s with a + search, 6.7 s for a search whose results the model read at length. +- Announcements heard on every slow tool, no audio glitch while the + stream waits for the tool. Straight into the answer it sounded rushed; + a 0.6 s pause after it fixed that (by ear). +- `agenda.py` matched khal's listing over 7 and 60 days on the real vdir, + runs in ~60 ms. + ### TTS (2026-09-25) `piper-tts` on CPU (onnxruntime), ~5 s sentence per voice: @@ -357,7 +393,8 @@ no particular order: reads the stored flag; confirm it is 1 while a game holds it on. - **Latency:** STT ~1.2 s is the biggest share; the double encoder pass (detect + transcribe) is the target if it matters. -- **Tools** (time, weather, calendar) so the model stops guessing facts. +- **Phase 2 tools:** tools that write or modify (events, messages, + desktop), own spec, after phase 1 has been used for a while. - OpenVINO in the whisper.cpp package: runs on CPU only here (no Intel compute-runtime), decision goes to `my-slackbuilds`. @@ -2,7 +2,8 @@ A local push-to-talk voice assistant. Unmute the microphone, speak, mute it: the assistant transcribes what you said, asks a local LLM, and speaks the -answer. Italian and English. Nothing leaves the machine. +answer. Italian and English. Speech, transcripts and the model stay on the +machine; only tools you enable (weather, web search) reach the network. ``` mic button ──> arecord ──> whisper-server (STT) ──> llama-server (LLM) ──> Kokoro (TTS) ──> aplay @@ -32,9 +33,11 @@ mic button ──> arecord ──> whisper-server (STT) ──> llama-server (LL - a running `llama-server` with an OpenAI-compatible API - Python 3 with `onnxruntime`, `numpy` and `kokoro-onnx` - `espeak-ng`, ALSA utils (`amixer`, `arecord`, `aplay`), `curl` +- `jq` for the weather and search example tools, `python3-dateutil` for the calendar one - a USB microphone with a hardware mute button exposed as an ALSA switch -Paths, the card name and ports are constants at the top of `assistant.py`. +Paths, the card name, ports and voices can be changed in `config.toml`, see +"Configuration and tools" below. ## Usage @@ -43,6 +46,40 @@ python3 assistant.py # run python3 assistant.py --test # self-check ``` +## Configuration and tools + +Everything is optional. Settings and tools live in +`~/.config/desktop-assistant/` (or `$XDG_CONFIG_HOME/desktop-assistant/`): + +```sh +mkdir -p ~/.config/desktop-assistant +cp examples/config.toml ~/.config/desktop-assistant/ +cp -r examples/tools ~/.config/desktop-assistant/ +``` + +- `config.toml`: every key is optional, the example lists the defaults. + `[env]` is added to every tool's environment; put instance URLs and + key file paths there. +- `tools/<name>.toml`: one tool each, enabled by being there. A tool is a + command run without a shell, its stdout goes back to the model: + +```toml +description = "What the tool does, for the model" +command = ["script.sh", "{param}"] # looked up in tools/ first, then PATH +timeout = 10 +announce.it = ["Cerco un attimo."] # optional, said while it runs +announce.en = ["Let me look that up."] + +[params.param] +type = "string" +description = "What the model should pass" +``` + +The example tools only read: `now` (date), `calendar` (`agenda.py` over +a vdirsyncer folder), `weather` (OpenWeatherMap, key file in the +conky-theme-udt format), `search` (a SearXNG instance with the JSON +format enabled). + ## License Copyright (C) 2026 Danilo M. <danix@danix.xyz> |