# Cloud models: design

Date: 2026-10-23. Status: implemented (2026-10-23), daemon and quickshell; live-checked.

## Goal

Let a chat use a cloud model instead of the one llama-server has loaded.
Providers and their models are set up in the quickshell settings window;
the canvas header's model name becomes a dropdown that picks the model
for that chat. The preferred model may be a cloud one.

This changes the project's rule "Everything stays local: no cloud
STT/TTS/LLM fallbacks". The new rule: no silent cloud fallback, ever. A
cloud model is used only when the user picks it for a chat; tools opt in
to cloud chats one by one; audio never leaves the machine.

## What leaves the machine in a cloud chat

The provider receives the system prompt, the date stamp, the history
window (10 turns, tool calls with placeholder results), the question,
attachments, and the results of the tools that opted in. That is the
privacy cost, and the reason tools default to staying out.

## Config

One `[[cloud]]` table per provider in `config.toml`:

```toml
[[cloud]]
name    = "openrouter"
url     = "https://openrouter.ai/api/v1"
key_cmd = "pass show llm/openrouter"
models  = ["mistralai/mistral-large", "deepseek/deepseek-chat"]
```

- Providers are OpenAI-compatible only: `GET {url}/models` and
  `POST {url}/chat/completions`, `Authorization: Bearer <key>`. That
  covers OpenRouter, Mistral, Groq, DeepSeek, OpenAI, and the
  OpenAI-compatible endpoints of Gemini and Anthropic.
- `key_cmd` is an argv string (split with `shlex`, run without a shell);
  its stdout, stripped, is the key. Run once per provider at startup. The
  key is never logged, never sent on the socket, never written by the
  settings page.
- `name` must be unique and contain no `:`. `merge()` today handles
  tables, not arrays of tables: `cloud` is validated on its own (list of
  tables, exactly the four keys, all strings except `models`, a list of
  strings). A typo fails loudly, as for every other key.
- `key_cmd` failing (non-zero exit, empty output, 10 s timeout) drops
  that provider: one log line, its models absent from `models`. The
  daemon still starts.

## Model ids

`"<provider>:<model>"` for a cloud model, the bare router id for a local
one. A prefix counts only when it names a configured provider, so a local
id containing `:` stays local. Split on the first `:` only: cloud model
ids contain `/` and sometimes `:` (OpenRouter's `:free`).

## Daemon

- **Per-chat model.** `conversations` gets a `model TEXT` column; an
  existing database gains it with `ALTER TABLE` when missing. `ask` takes
  an optional `model`; the first ask of a chat stores it, a different one
  updates it. NULL (old chats, asks without `model`) means today's
  behaviour: whatever the router has loaded. Opening a chat reports its
  model (`open` answers with it) so the canvas can set the dropdown.
- **Local model.** A chosen local id is sent to the router as is, and the
  router swaps it in if another is loaded. The old rule "the daemon never
  asks for a model" becomes "only for a model the user picked for this
  chat". `ctx` still carries the model actually answering.
- **Cloud model.** Same request body minus `chat_template_kwargs` (strict
  providers reject unknown fields). No `/props`: `ctx` max comes from the
  provider's `/models`, `max_model_len` (vLLM, Hetzner) or
  `context_length` (OpenRouter), read when the chat's model changes; 0
  when neither is there, and the canvas then shows used tokens only. A
  chat's attachment budget uses the same size. Reasoning deltas are read
  from `reasoning_content` (llama-server) or `reasoning` (OpenRouter).
  `temperature` from config applies when set. Vision and audio are
  unknown for cloud models (`/models` does not report them), so
  attachments are sent and a provider refusal comes back as `error`.
- **Tools.** A tool file may set `cloud = true`. A cloud chat is offered
  only those; the default is false. A local chat is offered all of them,
  as today. The example tools ship with the flag only where the output is
  not personal (`calc`, `now`, `search`, `weather`); mail, finance,
  calendar, playing, VMs, Bluetooth, updates, modes, clone and project
  stay out.
- **Voice.** Transcription always uses the local loaded model, whatever
  the chat's model. The audio never leaves; only the transcript does, as
  the text question. If the loaded model cannot hear, the ask is refused
  as today.
- **Titles.** Written by the local loaded model, thinking off, whatever
  the chat's model (changed after the live check: a cloud model reasoned
  over the title for ~12 s).
- **To .md** (`write`) has no chat, so it always uses the local loaded
  model.
- **`models` command.** Answers with the router's models (id, status)
  plus every configured cloud model, each with `cloud: true|false` and,
  for cloud, its provider name. Feeds the dropdown.
- **`--cloud-models <name>`.** CLI mode: reads `config.toml`, runs that
  provider's `key_cmd`, prints the provider's `/models` ids as JSON, one
  array, sorted, and exits. The settings page runs it to fetch the list,
  so the key stays in a short-lived daemon process and never reaches QML.
- **Errors.** Provider HTTP error: `error` event with status and body,
  capped at 2000 characters like tool errors, nothing written to history.
  Timeouts as for the router.

## Settings page (quickshell, `desktop/modules/assistant/`)

A "Cloud" section under "Model": one card per provider with name, URL,
key command, and Fetch. Fetch saves the draft, runs `assistant.py
--cloud-models <name>`, and shows the ids as a checklist; the checked
ones become `models`. Add and remove provider. Saved through the existing
line-by-line `Config.js` path, still parsed with `tomllib` before the
atomic replace. Restart offered as for every config change.

Prefer extends to cloud models: `Prefs.assistantModel` holds the
prefixed id. Opening the canvas with a cloud preference loads nothing
locally.

## Canvas header (quickshell, `desktop/assistant/`)

The model name becomes a dropdown from the `models` command: local
models, then cloud models grouped by provider, cloud ones marked with an
icon. A new chat starts on the preferred model; a reopened chat on its
stored one. Picking a model applies from the next ask.

Switching a chat that already has turns from local to cloud sends its
whole history window to the provider on the next ask, so the first such
switch in a chat shows one line: "History in this chat goes to
<provider>."

## Testing

`--test` gains a fake provider: a local `http.server` stub on a random
port, `key_cmd` an `echo`. It checks: prefix routing (cloud id to the
stub with the Bearer key, local id to the router stub), a local id with
`:` stays local, `cloud = false` tools absent from the cloud request,
no `chat_template_kwargs` in it, the `model` column stored and reported
on reopen, the `ALTER TABLE` on an old database, a failing `key_cmd`
dropping its provider, and the key absent from every socket event and
log line.

Live check after: one cloud chat with a tool-using question (`weather`),
one asking for mail (must answer that it has no tool for it), a voice
question in a cloud chat (transcribed locally), and switching a chat
local to cloud and back.

## Docs

AGENTS.md: the opening line and the "Everything stays local" agent rule
rewritten to the rule in Goal; a findings section after the live check.
README: config example for `[[cloud]]` and the `cloud = true` tool flag.

## Scope

Two repos, one plan each: the daemon here first (config, routing, tool
filter, `models`, `--cloud-models`, DB column), then quickshell (settings
section, dropdown). Out: non-OpenAI-compatible APIs, per-model
capability detection, cost tracking, cloud STT/TTS.