diff options
| author | Danilo M. <danix@danix.xyz> | 2026-10-19 11:20:12 +0200 |
|---|---|---|
| committer | Danilo M. <danix@danix.xyz> | 2026-10-19 11:20:12 +0200 |
| commit | c1fc2001219eb9aa349439b3007b57e1ced25335 (patch) | |
| tree | 900e3fbc9da3a58d8f4e08f68040dd080c75ef84 | |
| parent | 5ce0cdf77d19e74d85de8674e88b25db213e8534 (diff) | |
| download | quickshell-c1fc2001219eb9aa349439b3007b57e1ced25335.tar.gz quickshell-c1fc2001219eb9aa349439b3007b57e1ced25335.zip | |
assistant: a mic button asks by voice
Each recording is a pw-record Process of its own, made from a Component
like the Do row's runs, so a stopped recording's late exit cannot be
read as the next one's. Stopping sets running to false (SIGTERM), which
lets pw-record write the WAV header; the audio ask goes out from the
process's exit, not from the click.
Recording stops itself at 59 s: the daemon refuses over 60 s by file
size, and the stop lands a little after the tick. Escape cancels a
recording before it does anything else, and closing the card or losing
the daemon cancels it too. A pw-record that ends on its own says
"recording failed".
Checked: a scratch harness saw transcribing, thinking, idle for a 2 s
recording; by hand, every case in the spec passed with no pw-record
left behind, a desktop shell reload mid-recording included.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| -rw-r--r-- | desktop/assistant/Assistant.qml | 66 | ||||
| -rw-r--r-- | desktop/assistant/Card.qml | 24 | ||||
| -rw-r--r-- | desktop/assistant/README.md | 9 | ||||
| -rw-r--r-- | docs/superpowers/plans/2026-10-19-assistant-mic.md | 284 | ||||
| -rw-r--r-- | docs/superpowers/specs/2026-10-19-assistant-mic-design.md | 9 |
5 files changed, 385 insertions, 7 deletions
diff --git a/desktop/assistant/Assistant.qml b/desktop/assistant/Assistant.qml index 340c0cf..865f157 100644 --- a/desktop/assistant/Assistant.qml +++ b/desktop/assistant/Assistant.qml @@ -79,6 +79,7 @@ Scope { onConnectedChanged: if (!root.connected) { root.state = "idle"; root.runningTool = ""; + root.endRecording(false); if (root.opRunning && root.opDaemon) { root.opRunning = false; root.opError = "assistant gone"; } } @@ -107,10 +108,14 @@ Scope { root.open = true; } - function close() { root.open = false; root.historyOpen = false; root.pickerOpen = false; } - // Escape: the picker, then the history panel, then the card. + function close() { + root.endRecording(false); + root.open = false; root.historyOpen = false; root.pickerOpen = false; + } + // Escape: a recording, the picker, then the history panel, then the card. function back() { - if (root.pickerOpen) root.pickerOpen = false; + if (root.recording) root.endRecording(false); + else if (root.pickerOpen) root.pickerOpen = false; else if (root.historyOpen) root.historyOpen = false; else root.close(); } @@ -200,6 +205,61 @@ Scope { } } + // The mic: pw-record into one file, overwritten by each recording and + // never deleted (the runtime dir goes at logout). The daemon has the + // model transcribe it. Spec: docs/superpowers/specs/2026-10-19-assistant-mic-design.md. + readonly property string micPath: `${Quickshell.env("XDG_RUNTIME_DIR") || "/tmp"}/assistant-mic.wav` + property var mic: null // the recording's Process, null when not recording + property int micSecs: 0 + readonly property bool recording: root.mic !== null + + function record() { + if (root.recording || !root.connected || root.busy || root.opRunning || root.blocked) return; + root.notice = ""; + root.micSecs = 0; + root.mic = micRun.createObject(root); + } + + // SIGTERM: pw-record finishes the header, then exited sends the ask. + // ponytail: a recording started before the last one's exit shares the + // file; pw-record exits within milliseconds of the signal. + function endRecording(send) { + if (!root.mic) return; + root.mic.send = send; + root.mic.running = false; + root.mic = null; + } + + // 59, not 60: the daemon refuses over 60 s by file size, and the stop + // lands a little after the tick. + Timer { + running: root.recording + interval: 1000 + repeat: true + onTriggered: if (++root.micSecs >= 59) root.endRecording(true); + } + + Component { + id: micRun + Process { + id: rec + property bool send: false + running: true + command: ["pw-record", "--rate", "16000", "--channels", "1", "--format", "s16", root.micPath] + onExited: { + if (root.mic === rec) { // ended on its own: no mic, PipeWire down + root.mic = null; + root.notice = "recording failed"; + } else if (rec.send && root.connected) { + root.chatting = true; + root.send({ type: "ask", text: "", audio: root.micPath, + files: root.staged.map(f => f.path) }); + } + rec.destroy(); + } + } + } + Process { running: true command: ["xdg-user-dir", "DOCUMENTS"] diff --git a/desktop/assistant/Card.qml b/desktop/assistant/Card.qml index 4c76e64..d8826fa 100644 --- a/desktop/assistant/Card.qml +++ b/desktop/assistant/Card.qml @@ -431,10 +431,30 @@ Rectangle { } } + // nf-fa-microphone, checked in Inconsolata Nerd Font. Every state, + // entry included: SUPER+Home opens the card and the mic is one click away. + Button { + id: mic + anchors { bottom: parent.bottom; left: attach.right; leftMargin: attach.visible ? 8 : 0 } + height: box.height + icon: "" + text: card.canvas.recording + ? `${Math.floor(card.canvas.micSecs / 60)}:${String(card.canvas.micSecs % 60).padStart(2, "0")}` + : "" + danger: card.canvas.recording + enabled: card.canvas.connected && (card.canvas.recording + || (!card.canvas.busy && !card.canvas.opRunning && !card.canvas.blocked)) + onClicked: { + if (card.canvas.recording) card.canvas.endRecording(true); + else card.canvas.record(); + input.forceActiveFocus(); + } + } + Rectangle { id: box anchors { - bottom: parent.bottom; left: attach.right; leftMargin: attach.visible ? 8 : 0 + bottom: parent.bottom; left: mic.right; leftMargin: 8 right: history.left; rightMargin: 8 } // Grows with the text up to six lines, then scrolls. @@ -483,6 +503,8 @@ Rectangle { visible: input.text === "" elide: Text.ElideRight text: !card.canvas.connected ? "assistant not running" + : card.canvas.recording ? "Listening… click the mic to send" + : card.canvas.state === "transcribing" ? "Transcribing…" : card.mode === "entry" ? "How can I help you?" : card.canvas.staged.length ? `Ask about ${card.canvas.staged[0].name}…` : "Message the assistant" diff --git a/desktop/assistant/README.md b/desktop/assistant/README.md index 41c01b3..bd61cb4 100644 --- a/desktop/assistant/README.md +++ b/desktop/assistant/README.md @@ -32,8 +32,13 @@ dropped on the island (`../island/`) starts a new one in the file state. The paperclip opens a picker drawn in the card: a system dialog would open beneath the overlay that holds the keyboard. -Keys: Enter sends, Shift+Enter is a newline, Escape closes the picker, then -the history panel, then the card. Stop replaces Send while a reply streams. +The mic records with `pw-record` into `$XDG_RUNTIME_DIR/assistant-mic.wav` +until clicked again (at most 59 s), then asks with it: the daemon has the +model transcribe it and answers the words. + +Keys: Enter sends, Shift+Enter is a newline, Escape cancels a recording, +then closes the picker, the history panel, then the card. Stop replaces +Send while a reply streams. Notes: diff --git a/docs/superpowers/plans/2026-10-19-assistant-mic.md b/docs/superpowers/plans/2026-10-19-assistant-mic.md new file mode 100644 index 0000000..7917950 --- /dev/null +++ b/docs/superpowers/plans/2026-10-19-assistant-mic.md @@ -0,0 +1,284 @@ +# Assistant Mic Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** A mic button on the assistant canvas: click to record with `pw-record`, click again to ask the daemon with the recording. + +**Architecture:** `Assistant.qml` owns the recording: one `Process` per recording, created from a `Component` like the Do row's `opRun`, so a stopped recording's late exit cannot be mistaken for the next one. Stopping sets `running = false` (SIGTERM; pw-record then writes the WAV header) and the ask goes out from the process's `exited`. `Card.qml` only draws the button and the placeholder lines. Spec: `docs/superpowers/specs/2026-10-19-assistant-mic-design.md`; daemon side: desktop-assistant `docs/superpowers/specs/2026-10-19-voice-input-design.md` (implemented, running). + +**Tech Stack:** QML (Quickshell 0.3.1, `Quickshell.Io.Process`), PipeWire `pw-record`. + +QML here has no unit tests; the check is the scratch-harness load check the canvas and file-drop plans used, then the user's hand checks. + +--- + +### Task 1: Recording in `Assistant.qml` + +**Files:** +- Modify: `desktop/assistant/Assistant.qml` (`onConnectedChanged` ~line 79, `close()`/`back()` ~line 110, new block after the `opRun` Component ~line 199) + +- [ ] **Step 1: Stop a recording when the daemon goes away.** In `onConnectedChanged`, after `root.runningTool = "";`, add: + +```qml + root.endRecording(false); +``` + +- [ ] **Step 2: `close()` and `back()`.** Replace: + +```qml + function close() { root.open = false; root.historyOpen = false; root.pickerOpen = false; } + // Escape: the picker, then the history panel, then the card. + function back() { + if (root.pickerOpen) root.pickerOpen = false; +``` + +with: + +```qml + function close() { + root.endRecording(false); + root.open = false; root.historyOpen = false; root.pickerOpen = false; + } + // Escape: a recording, the picker, then the history panel, then the card. + function back() { + if (root.recording) root.endRecording(false); + else if (root.pickerOpen) root.pickerOpen = false; +``` + +- [ ] **Step 3: The recording.** After the `opRun` `Component { ... }` block, add: + +```qml + // The mic: pw-record into one file, overwritten by each recording and + // never deleted (the runtime dir goes at logout). The daemon has the + // model transcribe it. Spec: docs/superpowers/specs/2026-10-19-assistant-mic-design.md. + readonly property string micPath: `${Quickshell.env("XDG_RUNTIME_DIR") || "/tmp"}/assistant-mic.wav` + property var mic: null // the recording's Process, null when not recording + property int micSecs: 0 + readonly property bool recording: root.mic !== null + + function record() { + if (root.recording || !root.connected || root.busy || root.opRunning || root.blocked) return; + root.notice = ""; + root.micSecs = 0; + root.mic = micRun.createObject(root); + } + + // SIGTERM: pw-record finishes the header, then exited sends the ask. + // ponytail: a recording started before the last one's exit shares the + // file; pw-record exits within milliseconds of the signal. + function endRecording(send) { + if (!root.mic) return; + root.mic.send = send; + root.mic.running = false; + root.mic = null; + } + + // 59, not 60: the daemon refuses over 60 s by file size, and the stop + // lands a little after the tick. + Timer { + running: root.recording + interval: 1000 + repeat: true + onTriggered: if (++root.micSecs >= 59) root.endRecording(true); + } + + Component { + id: micRun + Process { + id: rec + property bool send: false + running: true + command: ["pw-record", "--rate", "16000", "--channels", "1", "--format", "s16", root.micPath] + onExited: { + if (root.mic === rec) { // ended on its own: no mic, PipeWire down + root.mic = null; + root.notice = "recording failed"; + } else if (rec.send && root.connected) { + root.chatting = true; + root.send({ type: "ask", text: "", audio: root.micPath, + files: root.staged.map(f => f.path) }); + } + rec.destroy(); + } + } + } +``` + +- [ ] **Step 4: Syntax check.** Run: `qmllint desktop/assistant/Assistant.qml 2>&1 | grep -i error || echo clean` (if `qmllint` is missing, the Task 3 load check catches errors). Expected: `clean`, or only the import warnings the file already had (compare with `git stash; qmllint ...; git stash pop`). + +--- + +### Task 2: The button in `Card.qml` + +**Files:** +- Modify: `desktop/assistant/Card.qml` (footer, ~lines 419-495) + +- [ ] **Step 1: The mic button.** After the paperclip `Button { id: attach ... }`, add: + +```qml + // nf-fa-microphone, checked in Inconsolata Nerd Font. Every state, + // entry included: SUPER+Home opens the card and the mic is one click away. + Button { + id: mic + anchors { bottom: parent.bottom; left: attach.right; leftMargin: attach.visible ? 8 : 0 } + height: box.height + icon: "" + text: card.canvas.recording + ? `${Math.floor(card.canvas.micSecs / 60)}:${String(card.canvas.micSecs % 60).padStart(2, "0")}` + : "" + danger: card.canvas.recording + enabled: card.canvas.connected && (card.canvas.recording + || (!card.canvas.busy && !card.canvas.opRunning && !card.canvas.blocked)) + onClicked: { + if (card.canvas.recording) card.canvas.endRecording(true); + else card.canvas.record(); + input.forceActiveFocus(); + } + } +``` + +- [ ] **Step 2: The input box sits after the mic.** In `Rectangle { id: box`, replace: + +```qml + bottom: parent.bottom; left: attach.right; leftMargin: attach.visible ? 8 : 0 +``` + +with: + +```qml + bottom: parent.bottom; left: mic.right; leftMargin: 8 +``` + +- [ ] **Step 3: Placeholder lines.** In the placeholder `Text`, replace: + +```qml + text: !card.canvas.connected ? "assistant not running" + : card.mode === "entry" ? "How can I help you?" +``` + +with: + +```qml + text: !card.canvas.connected ? "assistant not running" + : card.canvas.recording ? "Listening… click the mic to send" + : card.canvas.state === "transcribing" ? "Transcribing…" + : card.mode === "entry" ? "How can I help you?" +``` + +- [ ] **Step 4: Docs.** In `desktop/assistant/README.md`, replace: + +``` +Keys: Enter sends, Shift+Enter is a newline, Escape closes the picker, then +the history panel, then the card. Stop replaces Send while a reply streams. +``` + +with: + +``` +The mic records with `pw-record` into `$XDG_RUNTIME_DIR/assistant-mic.wav` +until clicked again (at most 59 s), then asks with it: the daemon has the +model transcribe it and answers the words. + +Keys: Enter sends, Shift+Enter is a newline, Escape cancels a recording, +then closes the picker, the history panel, then the card. Stop replaces +Send while a reply streams. +``` + +--- + +### Task 3: Load check against a scratch daemon + +The mic records the room for about 3 s; the clip goes only to the local model. A desktop shell restart is the user's (`./reload.sh desktop`), so this runs a separate `qs` instance. + +- [ ] **Step 1: Scratch daemon.** Run in the background (harness-owned): + +```bash +mkdir -p /tmp/claude-1000/dar +XDG_RUNTIME_DIR=/tmp/claude-1000/dar XDG_DATA_HOME=/tmp/claude-1000/dar/data \ + python3 <desktop-assistant>/assistant.py +``` + +Expected log line: `ready on /tmp/claude-1000/dar/desktop-assistant.sock`. + +- [ ] **Step 2: Harness.** Mirror `desktop/` with links, then write `shell.qml`: + +```bash +R=<this repo>; H=<scratchpad>/harness-mic +mkdir -p $H +ln -sf $R/shared/Theme.qml $H/Theme.qml +ln -sf $R/shared/Glow.qml $H/Glow.qml +ln -sf $R/shared/Notify.qml $H/Notify.qml +ln -sf $R/shared/NotifyConfig.js $H/NotifyConfig.js +ln -sf $R/desktop/Button.qml $H/Button.qml +ln -sfn $R/desktop/assistant $H/assistant +``` + +`$H/shell.qml`: + +```qml +import Quickshell +import Quickshell.Wayland +import QtQuick +import "assistant" + +ShellRoot { + Assistant { id: a; socketPath: "/tmp/claude-1000/dar/desktop-assistant.sock" } + + PanelWindow { + anchors { top: true; left: true } + implicitWidth: 1 + implicitHeight: 1 + color: "transparent" + mask: Region {} + exclusionMode: ExclusionMode.Ignore + WlrLayershell.keyboardFocus: WlrKeyboardFocus.None + Card { id: c; width: 760; fullHeight: 700; canvas: a; turnModel: a.turnsModel } + } + + property int step: 0 + property var states: [] + Connections { target: a; function onStateChanged() { states.push(a.state); } } + Timer { + interval: 500; repeat: true; running: true + onTriggered: { + if (!a.connected) return; + step++; + if (step === 1) { a.historyOpen = true; a.record(); console.log("PROTO cancel-start recording=" + a.recording); } + else if (step === 3) { a.back(); console.log("PROTO cancelled recording=" + a.recording + " history=" + a.historyOpen); a.historyOpen = false; } + else if (step === 5) { a.record(); console.log("PROTO send-start recording=" + a.recording); } + else if (step === 11) { a.endRecording(true); console.log("PROTO sent secs=" + a.micSecs); } + else if (step > 11 && !a.busy && (a.turnsModel.count || a.notice !== "")) { + console.log("PROTO done states=" + JSON.stringify(states) + " turns=" + a.turnsModel.count + + " notice=" + JSON.stringify(a.notice) + + " error=" + JSON.stringify(a.turnsModel.count ? a.turnsModel.get(0).error : "")); + step = -1000; + } + } + } +} +``` + +Run: `qs -p <scratchpad>/harness-mic` in the background, read its output for `PROTO` lines. + +Expected: +- `cancel-start recording=true`, `cancelled recording=false history=true` (Escape only cancelled; `open` stays false so the real card never shows and grabs the keyboard) +- `send-start recording=true`, `sent secs=3` +- `done states=[..."transcribing"...]`, with either one turn (the room's words, or a reply) or the daemon's `heard nothing` as the turn's error or notice +- no QML warnings or errors from `assistant/`. + +- [ ] **Step 3: No stray recorder.** `pgrep -a pw-record` prints nothing. Then stop both (`qs kill -p <scratchpad>/harness-mic`, the daemon by PID) and `pgrep -a pw-record` again. + +- [ ] **Step 4: Commit** (after asking the user): + +```bash +git add desktop/assistant/Assistant.qml desktop/assistant/Card.qml desktop/assistant/README.md +git commit -m "assistant: a mic button asks by voice" +``` + +Body: pw-record per recording in its own Process, stop by SIGTERM then ask on exit (the header), 59 s cap and why, Escape and closing cancel, and the scratch load check's result. + +--- + +### Task 4: The user's hand checks + +`./reload.sh desktop` from the user's terminal (new properties in a loaded file). Then the spec's list, `pgrep -a pw-record` empty after each: ask by voice, Escape twice, close mid-recording, past 59 s, reload mid-recording, Stop while transcribing, with a file staged. Afterwards set the spec's status to implemented with the date, and commit with the hand-check results. diff --git a/docs/superpowers/specs/2026-10-19-assistant-mic-design.md b/docs/superpowers/specs/2026-10-19-assistant-mic-design.md index 444c323..118ab08 100644 --- a/docs/superpowers/specs/2026-10-19-assistant-mic-design.md +++ b/docs/superpowers/specs/2026-10-19-assistant-mic-design.md @@ -1,6 +1,6 @@ # Assistant mic: design -Date: 2026-10-19. Status: proposed. +Date: 2026-10-19. Status: implemented and checked by hand (2026-10-19). ## Goal @@ -88,3 +88,10 @@ By hand, `pgrep -a pw-record` empty after each: - `./reload.sh desktop` while recording: no stray `pw-record`; - Stop while transcribing: the turn ends, idle; - with a file staged: the question goes with the file. + +Checked 2026-10-19, all passed, no `pw-record` left after any: a voice +question answered; Escape cancelled, a second Escape closed the card; +closing mid-recording sent nothing; a long recording stopped at 59 s and +was transcribed right; `./reload.sh desktop` mid-recording; Stop while +transcribing; a question about a staged file. The scratch harness also +saw `transcribing`, `thinking`, `idle` for a 2 s recording. |
