1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
|
# Cloud models: design
Date: 2026-10-23. Status: implemented (2026-10-23), daemon and quickshell; live-checked.
## Goal
Let a chat use a cloud model instead of the one llama-server has loaded.
Providers and their models are set up in the quickshell settings window;
the canvas header's model name becomes a dropdown that picks the model
for that chat. The preferred model may be a cloud one.
This changes the project's rule "Everything stays local: no cloud
STT/TTS/LLM fallbacks". The new rule: no silent cloud fallback, ever. A
cloud model is used only when the user picks it for a chat; tools opt in
to cloud chats one by one; audio never leaves the machine.
## What leaves the machine in a cloud chat
The provider receives the system prompt, the date stamp, the history
window (10 turns, tool calls with placeholder results), the question,
attachments, and the results of the tools that opted in. That is the
privacy cost, and the reason tools default to staying out.
## Config
One `[[cloud]]` table per provider in `config.toml`:
```toml
[[cloud]]
name = "openrouter"
url = "https://openrouter.ai/api/v1"
key_cmd = "pass show llm/openrouter"
models = ["mistralai/mistral-large", "deepseek/deepseek-chat"]
```
- Providers are OpenAI-compatible only: `GET {url}/models` and
`POST {url}/chat/completions`, `Authorization: Bearer <key>`. That
covers OpenRouter, Mistral, Groq, DeepSeek, OpenAI, and the
OpenAI-compatible endpoints of Gemini and Anthropic.
- `key_cmd` is an argv string (split with `shlex`, run without a shell);
its stdout, stripped, is the key. Run once per provider at startup. The
key is never logged, never sent on the socket, never written by the
settings page.
- `name` must be unique and contain no `:`. `merge()` today handles
tables, not arrays of tables: `cloud` is validated on its own (list of
tables, exactly the four keys, all strings except `models`, a list of
strings). A typo fails loudly, as for every other key.
- `key_cmd` failing (non-zero exit, empty output, 10 s timeout) drops
that provider: one log line, its models absent from `models`. The
daemon still starts.
## Model ids
`"<provider>:<model>"` for a cloud model, the bare router id for a local
one. A prefix counts only when it names a configured provider, so a local
id containing `:` stays local. Split on the first `:` only: cloud model
ids contain `/` and sometimes `:` (OpenRouter's `:free`).
## Daemon
- **Per-chat model.** `conversations` gets a `model TEXT` column; an
existing database gains it with `ALTER TABLE` when missing. `ask` takes
an optional `model`; the first ask of a chat stores it, a different one
updates it. NULL (old chats, asks without `model`) means today's
behaviour: whatever the router has loaded. Opening a chat reports its
model (`open` answers with it) so the canvas can set the dropdown.
- **Local model.** A chosen local id is sent to the router as is, and the
router swaps it in if another is loaded. The old rule "the daemon never
asks for a model" becomes "only for a model the user picked for this
chat". `ctx` still carries the model actually answering.
- **Cloud model.** Same request body minus `chat_template_kwargs` (strict
providers reject unknown fields). No `/props`: `ctx` max comes from the
provider's `/models`, `max_model_len` (vLLM, Hetzner) or
`context_length` (OpenRouter), read when the chat's model changes; 0
when neither is there, and the canvas then shows used tokens only. A
chat's attachment budget uses the same size. Reasoning deltas are read
from `reasoning_content` (llama-server) or `reasoning` (OpenRouter).
`temperature` from config applies when set. Vision and audio are
unknown for cloud models (`/models` does not report them), so
attachments are sent and a provider refusal comes back as `error`.
- **Tools.** A tool file may set `cloud = true`. A cloud chat is offered
only those; the default is false. A local chat is offered all of them,
as today. The example tools ship with the flag only where the output is
not personal (`calc`, `now`, `search`, `weather`); mail, finance,
calendar, playing, VMs, Bluetooth, updates, modes, clone and project
stay out.
- **Voice.** Transcription always uses the local loaded model, whatever
the chat's model. The audio never leaves; only the transcript does, as
the text question. If the loaded model cannot hear, the ask is refused
as today.
- **Titles.** Written by the local loaded model, thinking off, whatever
the chat's model (changed after the live check: a cloud model reasoned
over the title for ~12 s).
- **To .md** (`write`) has no chat, so it always uses the local loaded
model.
- **`models` command.** Answers with the router's models (id, status)
plus every configured cloud model, each with `cloud: true|false` and,
for cloud, its provider name. Feeds the dropdown.
- **`--cloud-models <name>`.** CLI mode: reads `config.toml`, runs that
provider's `key_cmd`, prints the provider's `/models` ids as JSON, one
array, sorted, and exits. The settings page runs it to fetch the list,
so the key stays in a short-lived daemon process and never reaches QML.
- **Errors.** Provider HTTP error: `error` event with status and body,
capped at 2000 characters like tool errors, nothing written to history.
Timeouts as for the router.
## Settings page (quickshell, `desktop/modules/assistant/`)
A "Cloud" section under "Model": one card per provider with name, URL,
key command, and Fetch. Fetch saves the draft, runs `assistant.py
--cloud-models <name>`, and shows the ids as a checklist; the checked
ones become `models`. Add and remove provider. Saved through the existing
line-by-line `Config.js` path, still parsed with `tomllib` before the
atomic replace. Restart offered as for every config change.
Prefer extends to cloud models: `Prefs.assistantModel` holds the
prefixed id. Opening the canvas with a cloud preference loads nothing
locally.
## Canvas header (quickshell, `desktop/assistant/`)
The model name becomes a dropdown from the `models` command: local
models, then cloud models grouped by provider, cloud ones marked with an
icon. A new chat starts on the preferred model; a reopened chat on its
stored one. Picking a model applies from the next ask.
Switching a chat that already has turns from local to cloud sends its
whole history window to the provider on the next ask, so the first such
switch in a chat shows one line: "History in this chat goes to
<provider>."
## Testing
`--test` gains a fake provider: a local `http.server` stub on a random
port, `key_cmd` an `echo`. It checks: prefix routing (cloud id to the
stub with the Bearer key, local id to the router stub), a local id with
`:` stays local, `cloud = false` tools absent from the cloud request,
no `chat_template_kwargs` in it, the `model` column stored and reported
on reopen, the `ALTER TABLE` on an old database, a failing `key_cmd`
dropping its provider, and the key absent from every socket event and
log line.
Live check after: one cloud chat with a tool-using question (`weather`),
one asking for mail (must answer that it has no tool for it), a voice
question in a cloud chat (transcribed locally), and switching a chat
local to cloud and back.
## Docs
AGENTS.md: the opening line and the "Everything stays local" agent rule
rewritten to the rule in Goal; a findings section after the live check.
README: config example for `[[cloud]]` and the `cloud = true` tool flag.
## Scope
Two repos, one plan each: the daemon here first (config, routing, tool
filter, `models`, `--cloud-models`, DB column), then quickshell (settings
section, dropdown). Out: non-OpenAI-compatible APIs, per-model
capability detection, cost tracking, cloud STT/TTS.
|
