1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
|
# Attachments: design
Date: 2026-10-16. Status: approved in brainstorming, not implemented.
## Goal
Let a question carry files. The quickshell canvas stages files dropped on
the island and sends their paths with `ask`; the daemon reads them, gives
the model their content, and keeps enough in the history for a reopened
conversation to still talk about them. The canvas side is specified in the
quickshell repo, `docs/superpowers/specs/2026-10-16-file-drop-design.md`.
This is piece 1 of three. Piece 2 is per-type model prompt presets
(summarize, describe, explain) and piece 3 local file operations (PDF to
markdown, merge, video to audio or GIF, trackcrop); each gets its own spec.
## Verified before designing
- Gemma4-12B on the router reports `input_modalities` of
`["text", "image", "audio"]` in `/v1/models`; Qwen3.8-9B reports
`["text"]`. The field is how vision is detected.
- `pdftotext` and `pdftoppm` (poppler 26.09.0) are installed.
- `tesseract` is installed with **no language data** (`/usr/share/tessdata`
lists 0 languages), so it cannot OCR. Scanned pages go to the model as
images instead, which Gemma reads.
- llamachat's attachment code (`llamachat/backend.py`) is the model:
`--- file: name ---` blocks for text, `image_url` data URLs for images,
truncation to a character budget. Read for ideas; this is a separate
implementation.
## Protocol
`ask` gains an optional `files`: a list of absolute paths, at most 8.
```json
{"type": "ask", "text": "What changed?", "files": ["/home/you/report.pdf"]}
```
The daemon runs as the user and reads the paths itself, so nothing is
copied over the socket. `text` may be empty when files are given; the
model then gets the files alone.
The `user` event and each shown turn gain `files`:
`[{name, kind, pages, truncated}]` (`pages` for a PDF, else absent), for
the canvas's chips.
Errors answer with `error` (with `turn`, since they end the reply before it
starts) and nothing is sent to the model:
- not an absolute path, missing, not a regular file, or unreadable;
- more than 8 files;
- an image over 20 MB;
- a kind chat does not take: "chat supports text, images and PDF";
- an image, or a PDF needing page images, for a model whose
`input_modalities` lacks `image`: "<model> cannot see images".
## Extraction
| kind | detected by | the model gets |
| --- | --- | --- |
| text | a `text/*` mime type, or a code suffix (llamachat's list) | the file as UTF-8 (errors replaced) in a `--- file: name ---` block |
| image | png, jpeg, webp, gif | an `image_url` part, a base64 data URL |
| pdf | `.pdf` | `pdftotext -layout` per page in one block; a page with no text is rendered with `pdftoppm -r 150 -png` and sent as an image, at most 4 such pages |
| other | anything else | refused, see above |
Text blocks share a budget: half the free context,
`(ctx_max - ctx_used) / 2` tokens at 3 characters per token. A file cut to
fit is marked `truncated` in its block header and its chip. With
`ctx_max` unknown (0), the budget is 24000 characters.
## History
`turns` gains a `files` column (JSON, the chip list), added with
`ALTER TABLE turns ADD COLUMN files TEXT` when `PRAGMA table_info` shows it
missing, so an existing `history.db` keeps working.
In `turns.messages`, extracted text is stored inline: a reopened
conversation still has the content. Images are stored as a reference part
`{"type": "image_ref", "name", "path", "sha256"}`. When history is sent to
the model, each reference is resolved: if the path still holds a file with
that hash, it becomes the `image_url` again; otherwise the text
`[image <name> is no longer available]`. Rendered PDF pages are stored the
same way, with the PDF's path, its hash and the page number, and re-rendered
on resolve.
## Testing
`python3 assistant.py --test` gains:
- classification of a text file, a code suffix, an image, a PDF and an
unknown file;
- a PDF built in the test with `magick` (one page of text, one blank), whose
extraction gives the text and one page image;
- truncation at a small budget, with the block marked;
- an image reference that resolves after a save and reload, and becomes the
unavailable text once the file changes;
- the vision error, against the fake server's `/models` reporting
`["text"]`;
- each refusal above.
|
