# Assistant overlay: design Date: 2026-09-26. Status: **superseded** by `2026-10-16-text-daemon-design.md`; the overlay and the voice pipeline it drew were removed (tag `voice-v1`). ## Goal A visual companion for the voice assistant. When the mic button is pressed, a transparent overlay appears with a circle: the assistant's name and how full the model's context is inside it, the running tool's icon and name when one runs, and a ring of sound waves around it that follows the real audio, in one colour while the user talks and another while the assistant replies. It fades out a configurable delay after the reply. A press the assistant ignores (presentation or gaming mode) gets a notification saying why, instead of an overlay. The assistant must keep working exactly as now with no overlay running. ## Verified before designing - Quickshell 0.3.1 is installed; `Quickshell.Io` has `Socket` with a `SplitParser`, which reads a line-based stream. - The quickshell repo (`../quickshell`) holds standalone components, each its own shell started from Hyprland's autostart. `shared/Theme.qml` already reads unified-desktop-theme's `~/.cache/udt/udt-palette.qml` and follows it live; the palette has `accent` and `green`. - llama-server reports the context size per model: `GET /props?model=` returns `default_generation_settings.n_ctx` (32768 for the resident Gemma). Tokens used per request already come back in the stream (`usage.prompt_tokens`). - Presentation mode turns DND on; the shell's notification daemon then holds back low and normal notifications (listed in its drawer) and pops only critical ones. `notify-send` is installed. ## Architecture ``` assistant.py ──JSON lines──> $XDG_RUNTIME_DIR/desktop-assistant.sock ──> quickshell assistant/ (server) (client, reconnects) ``` The assistant owns a Unix socket and broadcasts events to every connected client. With no client connected, events are dropped: the voice loop never waits on the UI. The overlay connects, retries every 2 s while the assistant is down, and restarts independently. Rejected: quickshell as the server (the assistant would depend on the UI being up); state files (fine for state, useless for a 30 Hz level stream). ## Protocol One JSON object per line, UTF-8. The socket file is created with mode 0600. Clients only read; anything they send is ignored. | `type` | Fields | When | |--------|--------|------| | `hello` | `name`, `position`, `hide_delay`, `user_color`, `assistant_color`, `ctx_max`, `state` | right after a client connects | | `state` | `state`, and `tool` when `state` is `tool` | on every change | | `ctx` | `used`, `max` | after every LLM round | | `level` | `who` (`user` or `assistant`), `value` (0 to 1) | ~30 per second while audio flows | States: | `state` | Entered when | |---------|--------------| | `listening` | the button goes on (not ignored), including a barge-in | | `transcribing` | the button goes off with a usable clip | | `thinking` | STT is done; also again after a tool round | | `tool` | a tool starts; `tool` names it. One event per tool | | `speaking` | the first audio of a reply starts playing | | `idle` | the reply's audio has finished, or the turn ended without one (stray press, blank audio, interrupted) | `speaking` can overlap `thinking` and `tool`: sentences play while later ones are still being generated, and the announcement plays while its tool runs. The state event reports the reply thread's stage; whether the assistant is audible is what `level` events with `who: assistant` show. The overlay treats them independently. `ctx_max` comes from `/props?model=`, read when the model id changes (the router can swap models); `0` when unknown. A client connecting mid-turn gets the current state in `hello`, then the stream. ## Assistant side (`assistant.py`) - **Socket thread.** Stdlib `socket` + `threading`. Accepts clients, keeps a list, `emit(event)` serialises once and sends to each; a client that errors is dropped. The socket file is removed on exit and replaced if stale at startup. - **User level.** While recording, a thread reads the RAW file's new bytes every 33 ms and emits the RMS of that window. - **Assistant level.** For each synthesised sentence the speaker computes an envelope (RMS per 33 ms frame) and queues the frames with their due time on a playback clock: the stream's start plus the lead silence plus the samples already written, divided by the sample rate. One emitter thread sends each frame at its due time. A barge-in drops the pending frames of the stale turn. `ponytail:` no correction for the sound server's own output latency; add a constant offset if the waves visibly lead the voice. - **Level scale.** `value = clamp((20*log10(rms/32768) + 60) / 60, 0, 1)`: -60 dBFS and below is 0, full scale is 1. Perceptual, so quiet speech still moves the ring. - **Ignored press.** `presenting()` returns why (`presentation`, `gaming` or nothing). An ignored press sends `notify-send -a desktop-assistant -u normal` with the reason, reusing the previous notification's id (`-p` prints it, `-r` replaces), so repeated presses during one game make one entry. No overlay events for an ignored press. - **Config.** New `[overlay]` table in `config.toml`, all optional: ```toml [overlay] name = "Assistant" # shown in the circle position = "center" # center, top, bottom, top-left, top-right, bottom-left, bottom-right hide_delay = 3.0 # seconds idle before it fades out user_color = "accent" # udt palette names assistant_color = "green" ``` These reach the overlay in `hello`, so the assistant's config is the one place to change them. ## Overlay side (`../quickshell/assistant/`) A new standalone component in the quickshell repo, built under that repo's AGENTS.md (keep-alive 1x1 window, `qs` process name, verification rules). Started from Hyprland's autostart like the others. - **Window.** A full-screen, transparent, click-through (`mask: Region {}`) layer-shell window on the overlay layer, on the Hyprland-focused monitor at the moment it appears (first screen as fallback). It never takes keyboard focus. - **Visibility.** Appears on `listening`. Fades out (opacity, ~200 ms) once `idle` has lasted `hide_delay` seconds with no assistant level above silence; any other state cancels the fade. Nothing on an ignored press, since the assistant sends nothing then. - **Circle.** Fixed size (~160 px), filled with a translucent `base`. Inside: `name`, and below it `used / max` (for example `1115 / 32768`), or only `used` when `max` is 0. In `tool` state, a tool icon and the tool name replace the context line until the state changes. - **Waves.** A ring of radial bars around the circle drawn with `Canvas` from a rolling history of the last ~64 level values, so the ring moves like a waveform. Bars from `user` levels use `user_color`, from `assistant` levels `assistant_color`, looked up by name in `Theme`, so a udt theme change recolours them live. Unknown names fall back to `accent`. - **Position.** From `hello.position`, with a fixed margin from the screen edges for the edge and corner positions. - **Reconnect.** A `Socket` on the path; while disconnected a 2 s `Timer` retries. A lost connection hides the overlay. ## Testing - `assistant.py --test` gains: the level scale (silence, a full-scale sine, a -30 dBFS sine), the envelope frame count for a known length, `presenting()` returning the reason, and `emit` with a connected test client reading back one line per event (a socketpair, no running assistant needed). - Protocol by hand: `socat - UNIX-CONNECT:$XDG_RUNTIME_DIR/desktop-assistant.sock` while asking a question shows the event sequence. - Overlay: visual, judged by the user (the quickshell repo's rule: for anything visual, ask). Checks: appears on press, waves follow the voice in `accent` and the reply in `green`, tool icon on a search, fade after `hide_delay`, nothing but a notification during a game, reconnects after an assistant restart. ## Out of scope A transcript or history view, clickable controls (talk, stop, clear), multiple overlays per monitor, sound-server latency calibration.