From 0185271050ff66d7ce14aa83a990035fc62e73e4 Mon Sep 17 00:00:00 2001 From: "Danilo M." Date: Fri, 16 Oct 2026 13:27:58 +0200 Subject: Docs: desktop-assistant is a text daemon now README and AGENTS.md describe the socket protocol and the history instead of the voice pipeline; the voice-era findings stay, marked as such, since a return to voice starts from them and from the voice-v1 tag. The overlay spec is marked superseded, the daemon spec records what changed while planning, and the example tools lose their spoken announcements. Co-Authored-By: Claude Sonnet 5.5 --- README.md | 102 ++++++++++++++++++++++++++++++++------------------------------ 1 file changed, 53 insertions(+), 49 deletions(-) (limited to 'README.md') diff --git a/README.md b/README.md index c6d6f2f..61c2aa8 100644 --- a/README.md +++ b/README.md @@ -1,55 +1,73 @@ # desktop-assistant -A local push-to-talk voice assistant. Unmute the microphone, speak, mute it: -the assistant transcribes what you said, asks a local LLM, and speaks the -answer. Italian and English. Speech, transcripts and the model stay on the -machine; only tools you enable (weather, web search) reach the network. +A local text assistant daemon. Questions arrive on a Unix socket, the +model already loaded in a local llama-server answers, calling read-only +tools when a question needs them, and the reply streams back token by +token. Every conversation is kept in a local SQLite history with +full-text search. The model and the history stay on the machine; only +tools you enable (weather, web search) reach the network. -``` -mic button ──> arecord ──> whisper-server (STT) ──> llama-server (LLM) ──> Kokoro (TTS) ──> aplay -``` +The front end is the assistant canvas of the quickshell desktop +(`desktop/assistant/` in the quickshell repo, on SUPER+Home). Anything +that speaks JSON lines on a Unix socket can drive it. -## How it works - -- **Push-to-talk** is the microphone's own mute button: the script listens to - the ALSA `Mic Capture Switch` (`amixer events`). Switch on starts a - recording, switch off ends it. -- **Speech to text**: whisper.cpp's `whisper-server` with the - `large-v3-turbo` model on the GPU (Vulkan). The script starts it if it is - not already running and stops it on exit. -- **LLM**: an OpenAI-compatible `llama-server` on `localhost:8181`. The - script uses whichever model the server already has loaded, so it never - swaps models under other clients. -- **Text to speech**: Kokoro-82M through `kokoro-onnx`, one voice per - language, chosen per sentence from the language of the reply. Speech - starts after the first sentence, while the rest is still being generated. -- **Calls**: while the desktop's presentation mode is on - (`statusctl presentation get` returns `1`), button presses are ignored, so - a call can use the microphone. +The voice version (push-to-talk, whisper, Kokoro) is the tag `voice-v1`. ## Requirements -- whisper.cpp (`whisper-server`) and a whisper ggml model -- a running `llama-server` with an OpenAI-compatible API -- Python 3 with `onnxruntime`, `numpy` and `kokoro-onnx` -- `espeak-ng`, ALSA utils (`amixer`, `arecord`, `aplay`), `curl` -- `jq` for the weather and search example tools, `python3-dateutil` for the calendar one -- a USB microphone with a hardware mute button exposed as an ALSA switch - -Paths, the card name, ports and voices can be changed in `config.toml`, see -"Configuration and tools" below. +- a running `llama-server` with an OpenAI-compatible API, router mode +- Python 3 (stdlib only; SQLite with FTS5) +- `curl` and `jq` for the weather and search example tools, + `python3-dateutil` for the calendar one ## Usage ```sh python3 assistant.py # run python3 assistant.py --test # self-check -./assistant.sh start # detached, with the overlay +./assistant.sh start # detached ./assistant.sh stop -./assistant.sh restart # stop a running one, start again +./assistant.sh restart ./assistant.sh status ``` +The daemon uses whichever model the router has loaded, so it never swaps +a model out under another client. + +Only one daemon serves the socket: a second one refuses to start while the +first answers. + +## Protocol + +JSON lines both ways on `$XDG_RUNTIME_DIR/desktop-assistant.sock` (mode +0600). Every connected client sees every event. + +In: `ask {text}`, `stop`, `new`, `list`, `open {id}`, `search {q}`, +`delete {id}`. + +Out: `hello` on connect (state, model, context use, the current +conversation and any reply in flight), then `user`, `think`, `token`, +`tool`, `done`, `state`, `ctx`, `error`, and the answers `conversations`, +`conversation` and `results`. The full table is in +`docs/superpowers/specs/2026-10-16-text-daemon-design.md`. + +Ids are integers; a malformed command or argument answers `error` and the +connection stays open. + +Try it by hand: + +```sh +socat - UNIX-CONNECT:$XDG_RUNTIME_DIR/desktop-assistant.sock +{"type":"ask","text":"What time is it?"} +``` + +## History + +`~/.local/share/desktop-assistant/history.db` (`$XDG_DATA_HOME` +honoured), directory 0700, file 0600. The model is sent the last 10 turns +of the open conversation. Tool results are stored as a placeholder, not +their output. Reasoning is shown live and not stored. + ## Configuration and tools Everything is optional. Settings and tools live in @@ -71,8 +89,6 @@ cp -r examples/tools ~/.config/desktop-assistant/ description = "What the tool does, for the model" command = ["script.sh", "{param}"] # looked up in tools/ first, then PATH timeout = 10 -announce.it = ["Cerco un attimo."] # optional, said while it runs -announce.en = ["Let me look that up."] [params.param] type = "string" @@ -99,18 +115,6 @@ The example tools only read: full markdown recap opened in `MD_EDITOR` (default `xdg-open`); `CHANGELOG` and `PKGDIR` in `[env]` override the mirror path and package database -## Overlay - -The assistant broadcasts its state, context use, the running tool and the -audio levels as JSON lines on `$XDG_RUNTIME_DIR/desktop-assistant.sock` -(mode 0600). The overlay is the `assistant/` component of the quickshell -repo: a circle with the name, `used / max` context and the tool, pulsing -dots while it transcribes or thinks, and a ring of ripples following your -voice and the reply. It fades out after the reply. Name, position, delay -and colours are the `[overlay]` table in `config.toml`. With no overlay -running the assistant works the same. A press ignored in presentation or -gaming mode sends a normal-urgency notification instead. - ## License Copyright (C) 2026 Danilo M. -- cgit v1.2.3