diff options
Diffstat (limited to 'docs/superpowers/specs/2026-09-26-tools-and-config-design.md')
| -rw-r--r-- | docs/superpowers/specs/2026-09-26-tools-and-config-design.md | 242 |
1 files changed, 242 insertions, 0 deletions
diff --git a/docs/superpowers/specs/2026-09-26-tools-and-config-design.md b/docs/superpowers/specs/2026-09-26-tools-and-config-design.md new file mode 100644 index 0000000..b0d8aeb --- /dev/null +++ b/docs/superpowers/specs/2026-09-26-tools-and-config-design.md @@ -0,0 +1,242 @@ +# Tools and config folder: design + +Date: 2026-09-26. Status: approved in brainstorming, not implemented. +Phase 1 of 2: read-only tools (see "Phasing"). + +## Goal + +Give the assistant tools (date/time, calendar, weather, web search) so it +stops guessing facts, and make it customizable: what it can do and how is +decided by files in a config folder, not by editing `assistant.py`. + +## Verified before designing + +- llama-server runs with `--jinja`; Gemma4-12B emits OpenAI-style + `tool_calls` in the stream. Probe: "Che ore sono?" called `now()` in + 0.63 s, "What's the weather like in Rome?" called `weather` with + `{"place": "Rome"}` in 0.4 s, arguments streamed in fragments that must be + concatenated per `index`. A general question called no tool. +- The calendar is a vdirsyncer vdir: plain `.ics` files, one directory per + calendar. Recurring events are mostly yearly; most `RRULE` lines in the + files belong to `VTIMEZONE` blocks, not events. +- `python3-dateutil` 2.9.0 is installed as its own package. +- The SearXNG instance answers `/search?q=...&format=json`. +- Weather in the sibling conky-theme-udt project comes from OpenWeatherMap: + key and home city in an env file outside any repo, current conditions + cached as JSON every 15 minutes. + +## Config folder + +`${XDG_CONFIG_HOME:-~/.config}/desktop-assistant/`, read once at startup +with stdlib `tomllib`. + +``` +desktop-assistant/ + config.toml optional, every key optional + tools/ + <name>.toml one tool each; the file name is the tool name + <helper> scripts the tools call, if any +``` + +A missing folder, missing `config.toml` or empty `tools/` is valid: the +assistant then behaves exactly as it does today, with no tools. A tool is +enabled by its file being present; there is no enable flag. + +### config.toml + +Replaces the constants at the top of `assistant.py`. The current values +become the built-in defaults, merged key by key under whatever the file sets. + +```toml +card = "Microphone" # ALSA card id of the mic +llm = "http://localhost:8181/v1" +system = "You are a voice assistant..." +tool_rounds = 3 # max tool round trips per question + +[whisper] +model = "/data/voice-models/whisper/ggml-large-v3-turbo-q5_0.bin" +device = 0 +port = 8182 + +[kokoro] +model = "/data/voice-models/kokoro/kokoro-v1.0.onnx" +voices = "/data/voice-models/kokoro/voices-v1.0.bin" + +[voice.it] # style name = weight, blended +lang = "it" +blend = { if_sara = 0.8, af_bella = 0.2 } + +[voice.en] +lang = "en-us" +blend = { af_heart = 1.0 } + +[env] # added to every tool's environment +# SEARXNG_URL = "http://searxng.example.org" +# WEATHER_ENV = "/home/you/.config/udt/weather.env" +``` + +`[env]` is how personal values (instance URLs, key file paths) reach the +tool scripts without appearing in the repo or in tool files. + +An unknown key is an error at startup, printed with its name: a typo must +not silently fall back to a default. + +### Tool file + +```toml +# tools/weather.toml +description = "Current weather and short forecast. Leave place empty for home." +command = ["weather.sh", "{place}", "{when}"] +timeout = 10 +announce.it = ["Controllo il meteo."] +announce.en = ["Checking the weather."] + +[params.place] +type = "string" +description = "City name, empty for home" + +[params.when] +type = "string" +description = "now, today or tomorrow" +``` + +Top-level keys must come before the `[params.*]` tables (TOML assigns keys +after a table header to that table). + +- `description`, `command`: required. `timeout` (seconds): default 10. +- `params`: optional; each has `type` (`string`, `integer`, `number`, + `boolean`) and `description`. Sent to the model as the JSON schema of the + function. No `required` list: every parameter is optional, missing ones + become empty strings. +- `announce.<lang>`: optional list of phrases, see below. + +## Running a tool + +- `command` is an argv list, run without a shell. Each `{name}` in an + element is replaced by the argument's string value. A value can therefore + never become a second argument or a shell command; a test covers a value + like `x; rm -rf ~`. +- `command[0]` without a slash is looked up in `tools/` first, then `PATH`. +- Environment: the assistant's own, plus `[env]`. +- stdout is returned to the model, stripped and cut to 2000 characters. +- A non-zero exit, a timeout or a failure to start returns + `error: <reason>` (with stderr's last line when there is one) as the tool + result. The model then says it could not find out; the assistant never + crashes on a tool. +- An unknown tool name or arguments that are not valid JSON also return + `error: ...`. + +## The loop + +`ask()` sends `tools` with every request when any are configured. While +streaming: + +- `delta.content` is spoken as now, sentence by sentence. +- `delta.tool_calls` fragments are collected per `index`: `id` and + `function.name` from the first fragment, `function.arguments` concatenated. + +When the stream ends with tool calls, the assistant message with its +`tool_calls` and one `{"role": "tool", "tool_call_id": ..., "content": ...}` +per call are appended to the turn's messages, and the request is repeated. +After `tool_rounds` rounds the request is sent without `tools`, which forces +a text answer. + +History across turns keeps only the user text and the final spoken reply, +as now. Tool calls and results live only inside the turn, which keeps the +20-message window for conversation. + +## Announcements + +When a round has tool calls, before running them, the assistant speaks one +phrase from the first called tool that has an `announce` entry for the +language whisper detected for the question. The phrase is picked with +`random.choice` so it does not repeat word for word. At most one +announcement per question, even across several rounds. Tools without +`announce` are silent. + +Rejected: speaking only if a tool runs past ~1 s. It needs a timer thread +and the phrase would land at an arbitrary point. + +## Example tools + +Shipped in the repo under `examples/`, copied by the user into the config +folder. They contain no personal values; those come from `[env]`. + +| Tool | Command | Network | Announce | +|------|---------|---------|----------| +| `now` | `date "+%A %d %B %Y, %H:%M"` | none | silent | +| `calendar` | `calendar.py {start} {days}` | none | silent | +| `weather` | `weather.sh {place} {when}` | OpenWeatherMap, except home "now" | yes | +| `search` | `search.sh {query}` | SearXNG | yes | + +### calendar.py + +Reads the vdir named by `CALENDAR_DIR` (default +`~/.local/share/calendars`). `start` is an ISO date (empty = today), +`days` a count (empty = 1). Prints one line per event in range, sorted: +`YYYY-MM-DD HH:MM summary` or `YYYY-MM-DD all day summary`, or +`no events`. + +Parsing is stdlib: unfold continuation lines, take each `VEVENT` block +(skipping `VTIMEZONE`), read `DTSTART`, `SUMMARY`, `RRULE`, `EXDATE`. +Recurrence goes through `dateutil.rrule.rrulestr` on the event's own +`DTSTART`/`RRULE`/`EXDATE` lines with `forceset=True`, so every RRULE form +and exception date is handled by dateutil. Times with `TZID` are converted +to local time. `RECURRENCE-ID` overrides (a moved single occurrence): the +override replaces that occurrence. Its own runnable check: `calendar.py +--test` against a generated temp vdir with a one-off, a yearly and an +EXDATE'd event. + +### weather.sh + +Sources the env file named by `WEATHER_ENV` (same format as +conky-theme-udt's: `KEY`, `CITY`, `COUNTRY`, `UNITS`). + +- Empty `place` and `when` = now: reads conky-theme-udt's cache + (`${XDG_CACHE_HOME:-~/.cache}/udt/weather.json`) if younger than 30 + minutes, no network. Otherwise fetches `/data/2.5/weather`. +- Another place, or `when` = today/tomorrow: `/data/2.5/weather` or the + free `/data/2.5/forecast` (3-hourly, filtered to the day). +- Prints a short plain-text summary (place, conditions, temperature, + min/max, wind) for the model to phrase. +- Uses `curl` and `jq`, both already used by conky-theme-udt. + +### search.sh + +`curl` SearXNG `/search?q=<query>&format=json` at `SEARXNG_URL`, `jq` to +the top 5 results as `title: snippet`. The model answers from those. + +## Testing + +`assistant.py --test` gains: + +- argv substitution, including an injection-shaped value that must stay one + argument; +- a tool that times out and one that exits non-zero both yield `error: ...`; +- assembling streamed `tool_calls` fragments into complete calls; +- config merge: defaults survive a partial file, an unknown key fails. + +Then a live run: time, a calendar question, home and remote weather, a +search, and a plain question that must call nothing. + +## Phasing + +This spec is **phase 1: read-only tools only**. Its purpose is to prove the +mechanism (config folder, tool files, the loop, announcements) on tools +that cannot do damage when the model calls them wrongly or with bad +arguments. Every example tool only reads. + +**Phase 2**, a separate spec after phase 1 is tested live: tools that +write, edit or otherwise change something (adding calendar events, +sending messages, controlling the desktop). That spec decides what such +tools need that read-only ones do not, for example a spoken confirmation +before running, a marker in the tool file that says a tool modifies +state, and what happens on a misheard command. Nothing in phase 1 builds +toward those ahead of time. + +## Out of scope for phase 1 + +Modifying tools (phase 2), contacts, barge-in. The loop does not stop a +user from putting a modifying command in a tool file: the config folder is +trusted like any user config, but phase 1 neither ships nor tests such a +tool. |