1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
|
# Assistant overlay: design
Date: 2026-09-26. Status: approved in brainstorming, not implemented.
## Goal
A visual companion for the voice assistant. When the mic button is
pressed, a transparent overlay appears with a circle: the assistant's
name and how full the model's context is inside it, the running tool's
icon and name when one runs, and a ring of sound waves around it that
follows the real audio, in one colour while the user talks and another
while the assistant replies. It fades out a configurable delay after the
reply. A press the assistant ignores (presentation or gaming mode) gets a
notification saying why, instead of an overlay.
The assistant must keep working exactly as now with no overlay running.
## Verified before designing
- Quickshell 0.3.1 is installed; `Quickshell.Io` has `Socket` with a
`SplitParser`, which reads a line-based stream.
- The quickshell repo (`../quickshell`) holds standalone components, each
its own shell started from Hyprland's autostart. `shared/Theme.qml`
already reads unified-desktop-theme's `~/.cache/udt/udt-palette.qml`
and follows it live; the palette has `accent` and `green`.
- llama-server reports the context size per model:
`GET /props?model=<id>` returns `default_generation_settings.n_ctx`
(32768 for the resident Gemma). Tokens used per request already come
back in the stream (`usage.prompt_tokens`).
- Presentation mode turns DND on; the shell's notification daemon then
holds back low and normal notifications (listed in its drawer) and pops
only critical ones. `notify-send` is installed.
## Architecture
```
assistant.py ──JSON lines──> $XDG_RUNTIME_DIR/desktop-assistant.sock ──> quickshell assistant/
(server) (client, reconnects)
```
The assistant owns a Unix socket and broadcasts events to every
connected client. With no client connected, events are dropped: the
voice loop never waits on the UI. The overlay connects, retries every
2 s while the assistant is down, and restarts independently.
Rejected: quickshell as the server (the assistant would depend on the UI
being up); state files (fine for state, useless for a 30 Hz level
stream).
## Protocol
One JSON object per line, UTF-8. The socket file is created with mode
0600. Clients only read; anything they send is ignored.
| `type` | Fields | When |
|--------|--------|------|
| `hello` | `name`, `position`, `hide_delay`, `user_color`, `assistant_color`, `ctx_max`, `state` | right after a client connects |
| `state` | `state`, and `tool` when `state` is `tool` | on every change |
| `ctx` | `used`, `max` | after every LLM round |
| `level` | `who` (`user` or `assistant`), `value` (0 to 1) | ~30 per second while audio flows |
States:
| `state` | Entered when |
|---------|--------------|
| `listening` | the button goes on (not ignored), including a barge-in |
| `transcribing` | the button goes off with a usable clip |
| `thinking` | STT is done; also again after a tool round |
| `tool` | a tool starts; `tool` names it. One event per tool |
| `speaking` | the first audio of a reply starts playing |
| `idle` | the reply's audio has finished, or the turn ended without one (stray press, blank audio, interrupted) |
`speaking` can overlap `thinking` and `tool`: sentences play while later
ones are still being generated, and the announcement plays while its
tool runs. The state event reports the reply thread's stage; whether the
assistant is audible is what `level` events with `who: assistant` show.
The overlay treats them independently.
`ctx_max` comes from `/props?model=<id>`, read when the model id changes
(the router can swap models); `0` when unknown.
A client connecting mid-turn gets the current state in `hello`, then the
stream.
## Assistant side (`assistant.py`)
- **Socket thread.** Stdlib `socket` + `threading`. Accepts clients, keeps
a list, `emit(event)` serialises once and sends to each; a client that
errors is dropped. The socket file is removed on exit and replaced if
stale at startup.
- **User level.** While recording, a thread reads the RAW file's new bytes
every 33 ms and emits the RMS of that window.
- **Assistant level.** For each synthesised sentence the speaker computes
an envelope (RMS per 33 ms frame) and queues the frames with their due
time on a playback clock: the stream's start plus the lead silence plus
the samples already written, divided by the sample rate. One emitter
thread sends each frame at its due time. A barge-in drops the pending
frames of the stale turn. `ponytail:` no correction for the sound
server's own output latency; add a constant offset if the waves visibly
lead the voice.
- **Level scale.** `value = clamp((20*log10(rms/32768) + 60) / 60, 0, 1)`:
-60 dBFS and below is 0, full scale is 1. Perceptual, so quiet speech
still moves the ring.
- **Ignored press.** `presenting()` returns why (`presentation`, `gaming`
or nothing). An ignored press sends `notify-send -a desktop-assistant
-u normal` with the reason, reusing the previous notification's id
(`-p` prints it, `-r` replaces), so repeated presses during one game
make one entry. No overlay events for an ignored press.
- **Config.** New `[overlay]` table in `config.toml`, all optional:
```toml
[overlay]
name = "Assistant" # shown in the circle
position = "center" # center, top, bottom, top-left, top-right, bottom-left, bottom-right
hide_delay = 3.0 # seconds idle before it fades out
user_color = "accent" # udt palette names
assistant_color = "green"
```
These reach the overlay in `hello`, so the assistant's config is the one
place to change them.
## Overlay side (`../quickshell/assistant/`)
A new standalone component in the quickshell repo, built under that
repo's AGENTS.md (keep-alive 1x1 window, `qs` process name, verification
rules). Started from Hyprland's autostart like the others.
- **Window.** A full-screen, transparent, click-through (`mask: Region
{}`) layer-shell window on the overlay layer, on the Hyprland-focused
monitor at the moment it appears (first screen as fallback). It never
takes keyboard focus.
- **Visibility.** Appears on `listening`. Fades out (opacity, ~200 ms)
once `idle` has lasted `hide_delay` seconds with no assistant level
above silence; any other state cancels the fade. Nothing on an ignored
press, since the assistant sends nothing then.
- **Circle.** Fixed size (~160 px), filled with a translucent `base`.
Inside: `name`, and below it `used / max` (for example `1115 / 32768`),
or only `used` when `max` is 0. In `tool` state, a tool icon and the
tool name replace the context line until the state changes.
- **Waves.** A ring of radial bars around the circle drawn with `Canvas`
from a rolling history of the last ~64 level values, so the ring
moves like a waveform. Bars from `user` levels use `user_color`, from
`assistant` levels `assistant_color`, looked up by name in `Theme`, so
a udt theme change recolours them live. Unknown names fall back to
`accent`.
- **Position.** From `hello.position`, with a fixed margin from the
screen edges for the edge and corner positions.
- **Reconnect.** A `Socket` on the path; while disconnected a 2 s `Timer`
retries. A lost connection hides the overlay.
## Testing
- `assistant.py --test` gains: the level scale (silence, a full-scale
sine, a -30 dBFS sine), the envelope frame count for a known length,
`presenting()` returning the reason, and `emit` with a connected test
client reading back one line per event (a socketpair, no running
assistant needed).
- Protocol by hand: `socat - UNIX-CONNECT:$XDG_RUNTIME_DIR/desktop-assistant.sock`
while asking a question shows the event sequence.
- Overlay: visual, judged by the user (the quickshell repo's rule: for
anything visual, ask). Checks: appears on press, waves follow the voice
in `accent` and the reply in `green`, tool icon on a search, fade after
`hide_delay`, nothing but a notification during a game, reconnects after
an assistant restart.
## Out of scope
A transcript or history view, clickable controls (talk, stop, clear),
multiple overlays per monitor, sound-server latency calibration.
|
