1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
|
# Tools and config folder: design
Date: 2026-09-26. Status: approved in brainstorming, not implemented.
Phase 1 of 2: read-only tools (see "Phasing").
## Goal
Give the assistant tools (date/time, calendar, weather, web search) so it
stops guessing facts, and make it customizable: what it can do and how is
decided by files in a config folder, not by editing `assistant.py`.
## Verified before designing
- llama-server runs with `--jinja`; Gemma4-12B emits OpenAI-style
`tool_calls` in the stream. Probe: "Che ore sono?" called `now()` in
0.63 s, "What's the weather like in Rome?" called `weather` with
`{"place": "Rome"}` in 0.4 s, arguments streamed in fragments that must be
concatenated per `index`. A general question called no tool.
- The calendar is a vdirsyncer vdir: plain `.ics` files, one directory per
calendar. Recurring events are mostly yearly; most `RRULE` lines in the
files belong to `VTIMEZONE` blocks, not events.
- `python3-dateutil` 2.9.0 is installed as its own package.
- The SearXNG instance answers `/search?q=...&format=json`.
- Weather in the sibling conky-theme-udt project comes from OpenWeatherMap:
key and home city in an env file outside any repo, current conditions
cached as JSON every 15 minutes.
## Config folder
`${XDG_CONFIG_HOME:-~/.config}/desktop-assistant/`, read once at startup
with stdlib `tomllib`.
```
desktop-assistant/
config.toml optional, every key optional
tools/
<name>.toml one tool each; the file name is the tool name
<helper> scripts the tools call, if any
```
A missing folder, missing `config.toml` or empty `tools/` is valid: the
assistant then behaves exactly as it does today, with no tools. A tool is
enabled by its file being present; there is no enable flag.
### config.toml
Replaces the constants at the top of `assistant.py`. The current values
become the built-in defaults, merged key by key under whatever the file sets.
```toml
card = "Microphone" # ALSA card id of the mic
llm = "http://localhost:8181/v1"
system = "You are a voice assistant..."
tool_rounds = 3 # max tool round trips per question
[whisper]
model = "/data/voice-models/whisper/ggml-large-v3-turbo-q5_0.bin"
device = 0
port = 8182
[kokoro]
model = "/data/voice-models/kokoro/kokoro-v1.0.onnx"
voices = "/data/voice-models/kokoro/voices-v1.0.bin"
[voice.it] # style name = weight, blended
lang = "it"
blend = { if_sara = 0.8, af_bella = 0.2 }
[voice.en]
lang = "en-us"
blend = { af_heart = 1.0 }
[env] # added to every tool's environment
# SEARXNG_URL = "http://searxng.example.org"
# WEATHER_ENV = "/home/you/.config/udt/weather.env"
```
`[env]` is how personal values (instance URLs, key file paths) reach the
tool scripts without appearing in the repo or in tool files.
An unknown key is an error at startup, printed with its name: a typo must
not silently fall back to a default.
### Tool file
```toml
# tools/weather.toml
description = "Current weather and short forecast. Leave place empty for home."
command = ["weather.sh", "{place}", "{when}"]
timeout = 10
announce.it = ["Controllo il meteo."]
announce.en = ["Checking the weather."]
[params.place]
type = "string"
description = "City name, empty for home"
[params.when]
type = "string"
description = "now, today or tomorrow"
```
Top-level keys must come before the `[params.*]` tables (TOML assigns keys
after a table header to that table).
- `description`, `command`: required. `timeout` (seconds): default 10.
- `params`: optional; each has `type` (`string`, `integer`, `number`,
`boolean`) and `description`. Sent to the model as the JSON schema of the
function. No `required` list: every parameter is optional, missing ones
become empty strings.
- `announce.<lang>`: optional list of phrases, see below.
## Running a tool
- `command` is an argv list, run without a shell. Each `{name}` in an
element is replaced by the argument's string value. A value can therefore
never become a second argument or a shell command; a test covers a value
like `x; rm -rf ~`.
- `command[0]` without a slash is looked up in `tools/` first, then `PATH`.
- Environment: the assistant's own, plus `[env]`.
- stdout is returned to the model, stripped and cut to 2000 characters.
- A non-zero exit, a timeout or a failure to start returns
`error: <reason>` (with stderr's last line when there is one) as the tool
result. The model then says it could not find out; the assistant never
crashes on a tool.
- An unknown tool name or arguments that are not valid JSON also return
`error: ...`.
## The loop
`ask()` sends `tools` with every request when any are configured. While
streaming:
- `delta.content` is spoken as now, sentence by sentence.
- `delta.tool_calls` fragments are collected per `index`: `id` and
`function.name` from the first fragment, `function.arguments` concatenated.
When the stream ends with tool calls, the assistant message with its
`tool_calls` and one `{"role": "tool", "tool_call_id": ..., "content": ...}`
per call are appended to the turn's messages, and the request is repeated.
After `tool_rounds` rounds the request is sent without `tools`, which forces
a text answer.
History across turns keeps only the user text and the final spoken reply,
as now. Tool calls and results live only inside the turn, which keeps the
20-message window for conversation.
## Announcements
When a round has tool calls, before running them, the assistant speaks one
phrase from the first called tool that has an `announce` entry for the
language whisper detected for the question. The phrase is picked with
`random.choice` so it does not repeat word for word. At most one
announcement per question, even across several rounds. Tools without
`announce` are silent.
Rejected: speaking only if a tool runs past ~1 s. It needs a timer thread
and the phrase would land at an arbitrary point.
## Example tools
Shipped in the repo under `examples/`, copied by the user into the config
folder. They contain no personal values; those come from `[env]`.
| Tool | Command | Network | Announce |
|------|---------|---------|----------|
| `now` | `date "+%A %d %B %Y, %H:%M"` | none | silent |
| `calendar` | `calendar.py {start} {days}` | none | silent |
| `weather` | `weather.sh {place} {when}` | OpenWeatherMap, except home "now" | yes |
| `search` | `search.sh {query}` | SearXNG | yes |
### calendar.py
Reads the vdir named by `CALENDAR_DIR` (default
`~/.local/share/calendars`). `start` is an ISO date (empty = today),
`days` a count (empty = 1). Prints one line per event in range, sorted:
`YYYY-MM-DD HH:MM summary` or `YYYY-MM-DD all day summary`, or
`no events`.
Parsing is stdlib: unfold continuation lines, take each `VEVENT` block
(skipping `VTIMEZONE`), read `DTSTART`, `SUMMARY`, `RRULE`, `EXDATE`.
Recurrence goes through `dateutil.rrule.rrulestr` on the event's own
`DTSTART`/`RRULE`/`EXDATE` lines with `forceset=True`, so every RRULE form
and exception date is handled by dateutil. Times with `TZID` are converted
to local time. `RECURRENCE-ID` overrides (a moved single occurrence): the
override replaces that occurrence. Its own runnable check: `calendar.py
--test` against a generated temp vdir with a one-off, a yearly and an
EXDATE'd event.
### weather.sh
Sources the env file named by `WEATHER_ENV` (same format as
conky-theme-udt's: `KEY`, `CITY`, `COUNTRY`, `UNITS`).
- Empty `place` and `when` = now: reads conky-theme-udt's cache
(`${XDG_CACHE_HOME:-~/.cache}/udt/weather.json`) if younger than 30
minutes, no network. Otherwise fetches `/data/2.5/weather`.
- Another place, or `when` = today/tomorrow: `/data/2.5/weather` or the
free `/data/2.5/forecast` (3-hourly, filtered to the day).
- Prints a short plain-text summary (place, conditions, temperature,
min/max, wind) for the model to phrase.
- Uses `curl` and `jq`, both already used by conky-theme-udt.
### search.sh
`curl` SearXNG `/search?q=<query>&format=json` at `SEARXNG_URL`, `jq` to
the top 5 results as `title: snippet`. The model answers from those.
## Testing
`assistant.py --test` gains:
- argv substitution, including an injection-shaped value that must stay one
argument;
- a tool that times out and one that exits non-zero both yield `error: ...`;
- assembling streamed `tool_calls` fragments into complete calls;
- config merge: defaults survive a partial file, an unknown key fails.
Then a live run: time, a calendar question, home and remote weather, a
search, and a plain question that must call nothing.
## Phasing
This spec is **phase 1: read-only tools only**. Its purpose is to prove the
mechanism (config folder, tool files, the loop, announcements) on tools
that cannot do damage when the model calls them wrongly or with bad
arguments. Every example tool only reads.
**Phase 2**, a separate spec after phase 1 is tested live: tools that
write, edit or otherwise change something (adding calendar events,
sending messages, controlling the desktop). That spec decides what such
tools need that read-only ones do not, for example a spoken confirmation
before running, a marker in the tool file that says a tool modifies
state, and what happens on a misheard command. Nothing in phase 1 builds
toward those ahead of time.
## Out of scope for phase 1
Modifying tools (phase 2), contacts, barge-in. The loop does not stop a
user from putting a modifying command in a tool file: the config folder is
trusted like any user config, but phase 1 neither ships nor tests such a
tool.
|
