diff options
Diffstat (limited to 'docs/superpowers/plans/2026-10-19-assistant-mic.md')
| -rw-r--r-- | docs/superpowers/plans/2026-10-19-assistant-mic.md | 284 |
1 files changed, 284 insertions, 0 deletions
diff --git a/docs/superpowers/plans/2026-10-19-assistant-mic.md b/docs/superpowers/plans/2026-10-19-assistant-mic.md new file mode 100644 index 0000000..7917950 --- /dev/null +++ b/docs/superpowers/plans/2026-10-19-assistant-mic.md @@ -0,0 +1,284 @@ +# Assistant Mic Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** A mic button on the assistant canvas: click to record with `pw-record`, click again to ask the daemon with the recording. + +**Architecture:** `Assistant.qml` owns the recording: one `Process` per recording, created from a `Component` like the Do row's `opRun`, so a stopped recording's late exit cannot be mistaken for the next one. Stopping sets `running = false` (SIGTERM; pw-record then writes the WAV header) and the ask goes out from the process's `exited`. `Card.qml` only draws the button and the placeholder lines. Spec: `docs/superpowers/specs/2026-10-19-assistant-mic-design.md`; daemon side: desktop-assistant `docs/superpowers/specs/2026-10-19-voice-input-design.md` (implemented, running). + +**Tech Stack:** QML (Quickshell 0.3.1, `Quickshell.Io.Process`), PipeWire `pw-record`. + +QML here has no unit tests; the check is the scratch-harness load check the canvas and file-drop plans used, then the user's hand checks. + +--- + +### Task 1: Recording in `Assistant.qml` + +**Files:** +- Modify: `desktop/assistant/Assistant.qml` (`onConnectedChanged` ~line 79, `close()`/`back()` ~line 110, new block after the `opRun` Component ~line 199) + +- [ ] **Step 1: Stop a recording when the daemon goes away.** In `onConnectedChanged`, after `root.runningTool = "";`, add: + +```qml + root.endRecording(false); +``` + +- [ ] **Step 2: `close()` and `back()`.** Replace: + +```qml + function close() { root.open = false; root.historyOpen = false; root.pickerOpen = false; } + // Escape: the picker, then the history panel, then the card. + function back() { + if (root.pickerOpen) root.pickerOpen = false; +``` + +with: + +```qml + function close() { + root.endRecording(false); + root.open = false; root.historyOpen = false; root.pickerOpen = false; + } + // Escape: a recording, the picker, then the history panel, then the card. + function back() { + if (root.recording) root.endRecording(false); + else if (root.pickerOpen) root.pickerOpen = false; +``` + +- [ ] **Step 3: The recording.** After the `opRun` `Component { ... }` block, add: + +```qml + // The mic: pw-record into one file, overwritten by each recording and + // never deleted (the runtime dir goes at logout). The daemon has the + // model transcribe it. Spec: docs/superpowers/specs/2026-10-19-assistant-mic-design.md. + readonly property string micPath: `${Quickshell.env("XDG_RUNTIME_DIR") || "/tmp"}/assistant-mic.wav` + property var mic: null // the recording's Process, null when not recording + property int micSecs: 0 + readonly property bool recording: root.mic !== null + + function record() { + if (root.recording || !root.connected || root.busy || root.opRunning || root.blocked) return; + root.notice = ""; + root.micSecs = 0; + root.mic = micRun.createObject(root); + } + + // SIGTERM: pw-record finishes the header, then exited sends the ask. + // ponytail: a recording started before the last one's exit shares the + // file; pw-record exits within milliseconds of the signal. + function endRecording(send) { + if (!root.mic) return; + root.mic.send = send; + root.mic.running = false; + root.mic = null; + } + + // 59, not 60: the daemon refuses over 60 s by file size, and the stop + // lands a little after the tick. + Timer { + running: root.recording + interval: 1000 + repeat: true + onTriggered: if (++root.micSecs >= 59) root.endRecording(true); + } + + Component { + id: micRun + Process { + id: rec + property bool send: false + running: true + command: ["pw-record", "--rate", "16000", "--channels", "1", "--format", "s16", root.micPath] + onExited: { + if (root.mic === rec) { // ended on its own: no mic, PipeWire down + root.mic = null; + root.notice = "recording failed"; + } else if (rec.send && root.connected) { + root.chatting = true; + root.send({ type: "ask", text: "", audio: root.micPath, + files: root.staged.map(f => f.path) }); + } + rec.destroy(); + } + } + } +``` + +- [ ] **Step 4: Syntax check.** Run: `qmllint desktop/assistant/Assistant.qml 2>&1 | grep -i error || echo clean` (if `qmllint` is missing, the Task 3 load check catches errors). Expected: `clean`, or only the import warnings the file already had (compare with `git stash; qmllint ...; git stash pop`). + +--- + +### Task 2: The button in `Card.qml` + +**Files:** +- Modify: `desktop/assistant/Card.qml` (footer, ~lines 419-495) + +- [ ] **Step 1: The mic button.** After the paperclip `Button { id: attach ... }`, add: + +```qml + // nf-fa-microphone, checked in Inconsolata Nerd Font. Every state, + // entry included: SUPER+Home opens the card and the mic is one click away. + Button { + id: mic + anchors { bottom: parent.bottom; left: attach.right; leftMargin: attach.visible ? 8 : 0 } + height: box.height + icon: "" + text: card.canvas.recording + ? `${Math.floor(card.canvas.micSecs / 60)}:${String(card.canvas.micSecs % 60).padStart(2, "0")}` + : "" + danger: card.canvas.recording + enabled: card.canvas.connected && (card.canvas.recording + || (!card.canvas.busy && !card.canvas.opRunning && !card.canvas.blocked)) + onClicked: { + if (card.canvas.recording) card.canvas.endRecording(true); + else card.canvas.record(); + input.forceActiveFocus(); + } + } +``` + +- [ ] **Step 2: The input box sits after the mic.** In `Rectangle { id: box`, replace: + +```qml + bottom: parent.bottom; left: attach.right; leftMargin: attach.visible ? 8 : 0 +``` + +with: + +```qml + bottom: parent.bottom; left: mic.right; leftMargin: 8 +``` + +- [ ] **Step 3: Placeholder lines.** In the placeholder `Text`, replace: + +```qml + text: !card.canvas.connected ? "assistant not running" + : card.mode === "entry" ? "How can I help you?" +``` + +with: + +```qml + text: !card.canvas.connected ? "assistant not running" + : card.canvas.recording ? "Listening… click the mic to send" + : card.canvas.state === "transcribing" ? "Transcribing…" + : card.mode === "entry" ? "How can I help you?" +``` + +- [ ] **Step 4: Docs.** In `desktop/assistant/README.md`, replace: + +``` +Keys: Enter sends, Shift+Enter is a newline, Escape closes the picker, then +the history panel, then the card. Stop replaces Send while a reply streams. +``` + +with: + +``` +The mic records with `pw-record` into `$XDG_RUNTIME_DIR/assistant-mic.wav` +until clicked again (at most 59 s), then asks with it: the daemon has the +model transcribe it and answers the words. + +Keys: Enter sends, Shift+Enter is a newline, Escape cancels a recording, +then closes the picker, the history panel, then the card. Stop replaces +Send while a reply streams. +``` + +--- + +### Task 3: Load check against a scratch daemon + +The mic records the room for about 3 s; the clip goes only to the local model. A desktop shell restart is the user's (`./reload.sh desktop`), so this runs a separate `qs` instance. + +- [ ] **Step 1: Scratch daemon.** Run in the background (harness-owned): + +```bash +mkdir -p /tmp/claude-1000/dar +XDG_RUNTIME_DIR=/tmp/claude-1000/dar XDG_DATA_HOME=/tmp/claude-1000/dar/data \ + python3 <desktop-assistant>/assistant.py +``` + +Expected log line: `ready on /tmp/claude-1000/dar/desktop-assistant.sock`. + +- [ ] **Step 2: Harness.** Mirror `desktop/` with links, then write `shell.qml`: + +```bash +R=<this repo>; H=<scratchpad>/harness-mic +mkdir -p $H +ln -sf $R/shared/Theme.qml $H/Theme.qml +ln -sf $R/shared/Glow.qml $H/Glow.qml +ln -sf $R/shared/Notify.qml $H/Notify.qml +ln -sf $R/shared/NotifyConfig.js $H/NotifyConfig.js +ln -sf $R/desktop/Button.qml $H/Button.qml +ln -sfn $R/desktop/assistant $H/assistant +``` + +`$H/shell.qml`: + +```qml +import Quickshell +import Quickshell.Wayland +import QtQuick +import "assistant" + +ShellRoot { + Assistant { id: a; socketPath: "/tmp/claude-1000/dar/desktop-assistant.sock" } + + PanelWindow { + anchors { top: true; left: true } + implicitWidth: 1 + implicitHeight: 1 + color: "transparent" + mask: Region {} + exclusionMode: ExclusionMode.Ignore + WlrLayershell.keyboardFocus: WlrKeyboardFocus.None + Card { id: c; width: 760; fullHeight: 700; canvas: a; turnModel: a.turnsModel } + } + + property int step: 0 + property var states: [] + Connections { target: a; function onStateChanged() { states.push(a.state); } } + Timer { + interval: 500; repeat: true; running: true + onTriggered: { + if (!a.connected) return; + step++; + if (step === 1) { a.historyOpen = true; a.record(); console.log("PROTO cancel-start recording=" + a.recording); } + else if (step === 3) { a.back(); console.log("PROTO cancelled recording=" + a.recording + " history=" + a.historyOpen); a.historyOpen = false; } + else if (step === 5) { a.record(); console.log("PROTO send-start recording=" + a.recording); } + else if (step === 11) { a.endRecording(true); console.log("PROTO sent secs=" + a.micSecs); } + else if (step > 11 && !a.busy && (a.turnsModel.count || a.notice !== "")) { + console.log("PROTO done states=" + JSON.stringify(states) + " turns=" + a.turnsModel.count + + " notice=" + JSON.stringify(a.notice) + + " error=" + JSON.stringify(a.turnsModel.count ? a.turnsModel.get(0).error : "")); + step = -1000; + } + } + } +} +``` + +Run: `qs -p <scratchpad>/harness-mic` in the background, read its output for `PROTO` lines. + +Expected: +- `cancel-start recording=true`, `cancelled recording=false history=true` (Escape only cancelled; `open` stays false so the real card never shows and grabs the keyboard) +- `send-start recording=true`, `sent secs=3` +- `done states=[..."transcribing"...]`, with either one turn (the room's words, or a reply) or the daemon's `heard nothing` as the turn's error or notice +- no QML warnings or errors from `assistant/`. + +- [ ] **Step 3: No stray recorder.** `pgrep -a pw-record` prints nothing. Then stop both (`qs kill -p <scratchpad>/harness-mic`, the daemon by PID) and `pgrep -a pw-record` again. + +- [ ] **Step 4: Commit** (after asking the user): + +```bash +git add desktop/assistant/Assistant.qml desktop/assistant/Card.qml desktop/assistant/README.md +git commit -m "assistant: a mic button asks by voice" +``` + +Body: pw-record per recording in its own Process, stop by SIGTERM then ask on exit (the header), 59 s cap and why, Escape and closing cancel, and the scratch load check's result. + +--- + +### Task 4: The user's hand checks + +`./reload.sh desktop` from the user's terminal (new properties in a loaded file). Then the spec's list, `pgrep -a pw-record` empty after each: ask by voice, Escape twice, close mid-recording, past 59 s, reload mid-recording, Stop while transcribing, with a file staged. Afterwards set the spec's status to implemented with the date, and commit with the hand-check results. |