diff options
Diffstat (limited to 'AGENTS.md')
| -rw-r--r-- | AGENTS.md | 100 |
1 files changed, 61 insertions, 39 deletions
@@ -1,42 +1,49 @@ # desktop-assistant -A local desktop voice assistant: speak, get a spoken answer, nothing leaves -the machine. Everything runs on already-packaged local tools; this repo is -the glue between them. +A local text assistant daemon: questions in and streamed replies out on a +Unix socket, answered by the model already loaded in llama-server, with +read-only tools and a SQLite history. Nothing leaves the machine except +what an enabled tool fetches. -Status: **first working loop** in `assistant.py` (2026-09-25). Mic button -on/off is push-to-talk, whisper large-v3-turbo on the Arc transcribes, -the resident llama-server model answers, Kokoro speaks it. Button-off to -first spoken word ~2 s. See "Findings" and "Next steps". +Status (2026-10-16): **text daemon**. The front end is the quickshell +canvas (`desktop/assistant/` in the quickshell repo, SUPER+Home). Spec: +`docs/superpowers/specs/2026-10-16-text-daemon-design.md`. -Tools (2026-09-26, phase 1 of 2): the model can call read-only tools -(time, calendar, weather, search) defined as shell-command TOML files in +The voice version is the tag `voice-v1`: push-to-talk on the mic's mute +button, whisper large-v3-turbo, Kokoro. The sections from "Hardware" +through "Kokoro-82M" describe that version and its measurements. They are +kept because a return to voice starts from them. + +Tools (2026-09-26): read-only shell-command TOML files in `~/.config/desktop-assistant/tools/`; `config.toml` there overrides the -built-in settings. Examples in `examples/`. STT, LLM and TTS stay local; -only the weather and search tools reach the network, and only when a -question needs them. Spec: `docs/superpowers/specs/2026-09-26-tools-and-config-design.md`. +built-in settings. Examples in `examples/`. Spec: +`docs/superpowers/specs/2026-09-26-tools-and-config-design.md`. -Run: `python3 assistant.py` (`--test` for the self-check), or `./assistant.sh restart` -to replace a running one (and its overlay) detached from the terminal. +Run: `python3 assistant.py` (`--test` for the self-check), or +`./assistant.sh restart` to replace a running one detached from the +terminal. --- ## Pipeline ``` -mic button ──> arecord ──> whisper-server (STT) ──> llama-server (LLM) ──> Kokoro (TTS) ──> aplay +socket (ask) ──> llama-server (LLM, tools) ──> socket (think, token, tool, done) + └─> history.db (SQLite, FTS5) ``` -| Stage | Package (installed) | Binary | Notes | -|-------|---------------------|--------|-------| -| STT | `whisper.cpp` 1.9.4 | `whisper-server` (also `whisper-stream`, `whisper-cli`) | SDL2 mic capture, ffmpeg decoding, Vulkan + OpenVINO backends | -| LLM | `llama.cpp-vulkan` 0.5.0 | `llama-server` | already running as a system service, see below | -| TTS | `python3-kokoro-onnx` 0.6.1 | (Python module) | uses system `python3-onnxruntime` and `espeak-ng`, no `espeakng-loader` | +- **Concurrency:** one reader thread per client, one reply thread per + question. One lock guards the conversation, the turn counter and the + database. `say()` checks the turn under it before each emit, and stop + bumps the turn under it, so a stopped reply cannot emit after its `done`. +- **One daemon:** `serve()` probe-connects first and refuses to start over a socket that answers; `main()` removes the socket only if it served it. +- **Commands** are type-checked before they run (ids must be `int`: SQLite would match `"1"` to row 1 while the Python comparison with the current id would not) and run under a try, so a failing command answers `error` instead of killing the client's reader. +- **Client sockets** have a 2 s timeout, not non-blocking mode: a large + `hello` would not fit a non-blocking send and would drop the client. +- **Search** quotes the query as one FTS5 phrase with a prefix star. The + self-check proves typed FTS syntax (`"`, `*`, `NOT`, `OR`, `)`) is + matched as text and raises nothing. -The three packages are in the `my-slackbuilds` repo -(`../slackbuilds-superrepo/my-slackbuilds/`). Package-level changes (build -options, backends, bumps) happen there, under that repo's `AGENTS.md`, not -here. `openvino` 2026.4.0 is also packaged there, solely to feed whisper.cpp. ## Hardware @@ -106,6 +113,26 @@ not import from it. ## Findings +### Text daemon, live check (2026-10-16) + +Model Gemma4-12B-qat-mtp, ctx 32768, preset reasoning-budget 4096. + +- Thinking on: `think` events arrive (1017 deltas for the symlink question; + the first after 2.8 s). Reasoning took about 21 s, so the first token came + at 23.8 s for a two-line question. This is why `thinking` became a config + key (default true). +- "What time is it?": no tool call; the date stamp in the prompt answered + it. First token 4.9 s. +- "How many unread mails do I have?": tool round in order `tool` mail + done=false, `state` tool, `tool` mail done=true (0.12 s later), `state` + thinking, tokens; first token 6.7 s. +- ctx used went 1166, 1247, 1275, 1603 across the three questions and the + tool round. +- `list` returned the one conversation, `search` "symlink" returned it with + a `**symlink**` snippet. +- History dir 700; `history.db`, `-wal` and `-shm` 600. +- SIGTERM logged `exit: SystemExit 0` and removed the socket. + ### Models (2026-09-25) Everything under `/data/voice-models/` (outside the repo, kept apart from @@ -645,22 +672,17 @@ kokoro-onnx GitHub release `model-files-v1.0`. ## Next steps -Steps 1-5 of the original plan are done (models, backend comparison, live -mic, TTS, glue); push-to-talk came for free from the mic button. Open, in -no particular order: - -- **Latency:** STT ~1.2 s is the biggest share; the double encoder pass - (detect + transcribe) is the target if it matters. +- **Canvas:** the quickshell side, `desktop/assistant/`, its own plan in + that repo. +- **Local memory:** `docs/superpowers/plans/2026-09-30-local-memory.md` + still applies; its tasks 1-4 target this daemon, the voice-specific + bits of task 4 aside. +- **llamachat ports**, if the canvas replaces llamachat for good: + attachments, skills, a model picker, model-written titles. - **Phase 2 tools:** tools that write or modify (events, messages, - desktop), own spec, after phase 1 has been used for a while. -- OpenVINO in the whisper.cpp package: runs on CPU only here (no Intel - compute-runtime), decision goes to `my-slackbuilds`. -- **Wake word:** `hey_jarvis` works in quiet rooms, fails under loud - music (3/8); no echo canceller in pipewire. Pick an option from - "Wake word trial" before building anything. - -Open questions: whether an audio-capable LLM makes whisper unnecessary. -License: GPLv2 only (chosen 2026-09-25). + desktop), with a review step, own spec. +- **Voice:** start from `voice-v1`. The open items from that era (STT + latency, wake word, OpenVINO) are in the sections above. ## Package-side leftovers (tracked in my-slackbuilds, not here) |