1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
|
# desktop-assistant
A local push-to-talk voice assistant. Unmute the microphone, speak, mute it:
the assistant transcribes what you said, asks a local LLM, and speaks the
answer. Italian and English. Speech, transcripts and the model stay on the
machine; only tools you enable (weather, web search) reach the network.
```
mic button ──> arecord ──> whisper-server (STT) ──> llama-server (LLM) ──> Kokoro (TTS) ──> aplay
```
## How it works
- **Push-to-talk** is the microphone's own mute button: the script listens to
the ALSA `Mic Capture Switch` (`amixer events`). Switch on starts a
recording, switch off ends it.
- **Speech to text**: whisper.cpp's `whisper-server` with the
`large-v3-turbo` model on the GPU (Vulkan). The script starts it if it is
not already running and stops it on exit.
- **LLM**: an OpenAI-compatible `llama-server` on `localhost:8181`. The
script uses whichever model the server already has loaded, so it never
swaps models under other clients.
- **Text to speech**: Kokoro-82M through `kokoro-onnx`, one voice per
language, chosen per sentence from the language of the reply. Speech
starts after the first sentence, while the rest is still being generated.
- **Calls**: while the desktop's presentation mode is on
(`statusctl presentation get` returns `1`), button presses are ignored, so
a call can use the microphone.
## Requirements
- whisper.cpp (`whisper-server`) and a whisper ggml model
- a running `llama-server` with an OpenAI-compatible API
- Python 3 with `onnxruntime`, `numpy` and `kokoro-onnx`
- `espeak-ng`, ALSA utils (`amixer`, `arecord`, `aplay`), `curl`
- `jq` for the weather and search example tools, `python3-dateutil` for the calendar one
- a USB microphone with a hardware mute button exposed as an ALSA switch
Paths, the card name, ports and voices can be changed in `config.toml`, see
"Configuration and tools" below.
## Usage
```sh
python3 assistant.py # run
python3 assistant.py --test # self-check
```
## Configuration and tools
Everything is optional. Settings and tools live in
`~/.config/desktop-assistant/` (or `$XDG_CONFIG_HOME/desktop-assistant/`):
```sh
mkdir -p ~/.config/desktop-assistant
cp examples/config.toml ~/.config/desktop-assistant/
cp -r examples/tools ~/.config/desktop-assistant/
```
- `config.toml`: every key is optional, the example lists the defaults.
`[env]` is added to every tool's environment; put instance URLs and
key file paths there.
- `tools/<name>.toml`: one tool each, enabled by being there. A tool is a
command run without a shell, its stdout goes back to the model:
```toml
description = "What the tool does, for the model"
command = ["script.sh", "{param}"] # looked up in tools/ first, then PATH
timeout = 10
announce.it = ["Cerco un attimo."] # optional, said while it runs
announce.en = ["Let me look that up."]
[params.param]
type = "string"
description = "What the model should pass"
```
The example tools only read:
- `now`: date and time
- `calendar`: `agenda.py` over a vdirsyncer folder
- `weather`: OpenWeatherMap, key file in the conky-theme-udt format
- `search`: a SearXNG instance with the JSON format enabled
- `calc`: `qalc` (arithmetic, units, currencies, date math)
- `mail`: unread threads through `mailctl` searches, no tagging or sync
- `playing`: the current track from `playerctl`, or `mpc`
- `finance`: balances, budgets and spending from Firefly III's `firefly`
list commands
## License
Copyright (C) 2026 Danilo M. <danix@danix.xyz>
This program is free software; you can redistribute it and/or modify it
under the terms of the GNU General Public License version 2 as published by
the Free Software Foundation. See [LICENSE](LICENSE).
## Development Approach
This project is developed using AI-assisted tools. Code is generated with the help of AI based on human-provided specifications, design decisions, and iterative feedback.
All contributions are reviewed, tested, and curated by the maintainer before being included in the codebase. AI is used as a productivity and exploration tool, while human oversight remains central to all decisions.
The goal is to combine the flexibility of AI-assisted development with standard open-source practices such as transparency, review, and accountability.
|
