docs: spec for external cloud providers

Design for talking to OpenAI-compatible cloud providers alongside the
local router. Providers are config table entries, local becomes the
"default" one, and a bare base_url still synthesizes it so existing
configs keep working.

Cloud models carry no presets.ini entry and cost money, so missing
metadata (context size, vision, prices) is entered through a per-model
dialog stored in models.ini, and an approximate per-conversation cost
sits next to the context meter. Token counts and the producing model
move onto the messages table so cost survives reopening a chat and
prices each reply at whatever produced it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

1 files changed, 275 insertions, 0 deletions

diff --git a/docs/superpowers/specs/2026-08-09-external-providers-design.md b/docs/superpowers/specs/2026-08-09-external-providers-design.md
new file mode 100644
index 0000000..1174ded
--- /dev/null
+++ b/docs/superpowers/specs/2026-08-09-external-providers-design.md
@@ -0,0 +1,275 @@
+# External cloud providers
+
+Date: 2026-08-09
+
+## Goal
+
+Let llamachat talk to OpenAI-compatible cloud providers (together.ai,
+siliconflow) alongside the local llama.cpp router, without the local path
+losing any of its current behaviour and without the existing config file
+needing to be edited.
+
+Cloud models cost money and carry no `presets.ini` entry, so two problems
+come with them: the client has no context size or vision flag to work from,
+and every turn resends the whole conversation at a real price. Both are
+addressed by making the missing metadata enterable through a dialog and by
+putting an approximate cost next to the context meter.
+
+## Providers
+
+`config.toml` grows a `[providers.*]` table. The local router stops being
+special and becomes an ordinary entry named `default`.
+
+```toml
+[providers.default]
+base_url = "http://localhost:8181"
+
+[providers.together]
+base_url = "https://api.together.xyz"
+api_key = "pass:api/together"
+filter = ["qwen", "deepseek"]
+ctx_size = 32768
+price_in = 0.60
+price_out = 0.60
+```
+
+Per-provider keys, all optional except `base_url`:
+
+| Key | Meaning |
+| --- | --- |
+| `base_url` | Required. API root, `/v1/...` is appended as today. |
+| `api_key` | Absent means no `Authorization` header is sent. |
+| `filter` | List of case-insensitive substrings. Empty or absent shows every model. |
+| `ctx_size` | Prefill for the model dialog. Not applied to requests. |
+| `vision` | Prefill for the model dialog. |
+| `price_in` | USD per million input tokens. Prefill. |
+| `price_out` | USD per million output tokens. Prefill. |
+
+### Backward compatibility
+
+A top-level `base_url` with no `[providers]` table synthesizes
+`[providers.default]` from it. An existing config keeps working untouched
+and nothing needs migrating. If both a top-level `base_url` and an explicit
+`[providers.default]` are present, the explicit one wins.
+
+### Model identity
+
+Models are addressed as `provider:model`. The `default` provider is the
+exception: its models are shown and stored bare, so the local dropdown reads
+exactly as it does now and existing sessions keep resolving.
+
+A model id containing a colon is therefore ambiguous only in principle; the
+split is on the first colon, and any provider name that is not configured
+falls back to treating the whole string as a `default` model. That keeps an
+old session pointing at a local model working even if the model name itself
+grows a colon.
+
+### Filtering
+
+Filtering applies to cloud providers so the dropdown stays usable:
+together.ai lists roughly 200 models. A model is kept when any filter
+substring appears in its id, compared case-insensitively. `"qwen"` matches
+`Qwen/Qwen2.5-72B-Instruct-Turbo`.
+
+The `default` provider is never filtered.
+
+When a provider's filter matches nothing, the status bar reports it by name
+with the counts, e.g. `together: 0 of 213 models matched filter`. The
+provider contributes no models rather than silently falling back to all of
+them.
+
+A provider that cannot be reached does not break the model list: its models
+are omitted and the failure is reported in the status bar. Local models must
+remain selectable when the network is down.
+
+## API keys
+
+`api_key` is a single field, dispatched on prefix:
+
+| Form | Behaviour |
+| --- | --- |
+| `pass:NAME` | Runs `pass show NAME`, takes the first line. |
+| `env:VAR` | Reads the environment variable. |
+| anything else | Used literally. |
+
+Resolution is lazy: nothing runs until the first request to that provider,
+so a session that only touches local models never invokes `pass` and never
+triggers a pinentry. The resolved value is cached in memory for the process
+lifetime and never written to disk.
+
+The `pass` subprocess gets a 30 second timeout. A stuck pinentry then
+surfaces as a backend error naming the provider instead of freezing the
+worker thread. A missing entry, a non-zero exit, or empty output are all
+reported the same way.
+
+A literal key in `config.toml` works but the generated comments point at
+`pass:` as the better option.
+
+## Model metadata
+
+Three layers resolve each of `ctx_size`, `vision`, `price_in`, `price_out`,
+most specific first:
+
+1. `models.ini` entry for that exact `provider:model`
+2. `[providers.X]` default
+3. unknown
+
+Nothing has to be set in `config.toml`. The provider block is a convenience
+that prefills the dialog; the dialog alone is enough.
+
+Unknown degrades per value:
+
+| Value | Unset behaviour |
+| --- | --- |
+| `ctx_size` | Meter shows tokens used with no limit and no percentage. Attachment budget uses a fallback. |
+| `vision` | Treated as unknown, not as false: attaching an image is permitted and no dialog offers to switch models. |
+| `price_in` / `price_out` | No cost figure. Shown as blank for `default`, `?` for a provider that has an `api_key`. |
+
+The `?` distinguishes "local, free" from "cloud, price not entered". Both
+would otherwise render as an empty label, which reads as free.
+
+### models.ini
+
+Written by the app, beside `config.toml` and `state.ini`, parsed with
+configparser. Sections are keyed by full model id.
+
+```ini
+[together:Qwen/Qwen2.5-72B-Instruct-Turbo]
+ctx_size = 32768
+vision = false
+price_in = 1.20
+price_out = 1.20
+
+[together:meta-llama/Llama-Vision-Free]
+configured = false
+```
+
+`configured = false` records a cancelled dialog: the model was offered and
+skipped, so it is never offered again. That is distinct from an absent
+section, which means never asked.
+
+The file grows one section per cloud model touched, including abandoned
+ones. At the expected scale that is tens of entries in a plain text file the
+user can prune by hand, so no cleanup logic is written.
+
+### Model dialog
+
+Four fields, prefilled from the provider defaults: context size, a vision
+checkbox, input price, output price. Prices are per million tokens. Empty
+price fields are valid and mean unpriced.
+
+Two triggers:
+
+- **First selection.** Choosing a model in the dropdown that has no
+  `models.ini` section and belongs to a provider other than `default` opens
+  the dialog once, before anything is typed. Cancel writes
+  `configured = false`.
+- **On demand.** A button beside the model dropdown opens the dialog for the
+  current model, including local ones, and regardless of any previous
+  cancel. This is how a wrong value gets corrected.
+
+Local models are exempt from the automatic trigger because `presets.ini`
+already answers context and vision for them and they cost nothing. The
+on-demand path still reaches them, and a `models.ini` entry overrides the
+preset when one exists.
+
+## Cost
+
+### Storage
+
+`messages` gains three nullable columns:
+
+| Column | Filled |
+| --- | --- |
+| `prompt_tokens` | On assistant rows, from the usage chunk. |
+| `completion_tokens` | On assistant rows, from the usage chunk. |
+| `model` | On assistant rows, the model that produced the reply. |
+
+The migration is additive. Existing rows read as NULL and contribute zero,
+so old conversations show a cost of zero rather than a fabricated one.
+
+`model` is needed on the message rather than the session because `sessions`
+records only the current model. A conversation that switched models would
+otherwise be priced entirely at whichever model happens to be selected, and
+switching mid-conversation is exactly what a mixed local-and-cloud setup
+invites.
+
+### Readout
+
+Beside the context meter: the conversation's cost so far, plus what sending
+the current draft would cost, shown separately so the two are not conflated:
+
+```
+$0.043 +$0.011
+```
+
+Cost so far is summed from the stored token counts, pricing each assistant
+message by its own `model`, so it survives reopening a conversation. The
+projected figure reuses the token estimate the context meter already
+computes for the draft, priced at the currently selected model's input rate.
+
+For an unpriced cloud model the figure is `?`. For a local model the label is
+empty.
+
+### Why the projection matters
+
+`_chat_context` rebuilds every request from all stored rows, so reopening a
+long conversation and sending one message resends the entire history. That
+is free locally and billed per turn on a cloud provider. Showing the
+projected cost of the next send is what makes that visible before it is
+paid rather than after.
+
+No trimming, summarization or context management is part of this work.
+Visibility is the whole intervention: an expensive conversation is apparent,
+and starting a new one is the user's call.
+
+## Out of scope
+
+- History trimming or summarization for long cloud conversations.
+- A lifetime or per-month spend total. The stored counts make it a `SUM`
+  away if it is ever wanted.
+- Per-model tables in `config.toml`. The dialog covers that ground.
+- Streaming or tool-call behaviour differences between providers, beyond not
+  assuming they are absent. See the risk below.
+
+## Risks
+
+**Tool calling and reasoning content are unverified on cloud providers.**
+The web search loop depends on tool-call fragments arriving in the delta's
+`tool_calls` field and a `finish_reason` of `tool_calls`, and the reasoning
+pane depends on `reasoning_content` deltas. Both are llama.cpp behaviours
+that cloud providers implement with variations. Search on a cloud model may
+need per-provider work once it is tried against a real endpoint. Nothing in
+this design assumes it works out of the box, and local search behaviour must
+not regress.
+
+**A wrong price entered once is silently wrong forever.** The cost figure is
+only as good as the numbers typed into the dialog, and no API publishes them
+in a standard form. The readout is explicitly an approximation.
+
+**Provider outages must not break the local path.** Model listing, key
+resolution and cost display all have to fail soft, leaving local models
+usable.
+
+## Testing
+
+Following the project's existing test approach, written before the
+implementation:
+
+- Config: providers parsed, bare `base_url` synthesizing `default`, explicit
+  `default` winning over it, filter matching case-insensitively, empty
+  filter meaning all, `default` never filtered.
+- Key resolution: each prefix form, lazy resolution not firing for local
+  requests, caching after first resolution, timeout and failure paths
+  reported as errors.
+- Model ids: split on first colon, unknown provider falling back to
+  `default`, `default` models rendering bare.
+- Metadata resolution: the three layers in order, unknown values degrading
+  as tabulated, `configured = false` suppressing the dialog.
+- `models.ini` round trip: written, reread, values preserved.
+- Cost: summing per message by that message's model, NULL rows counting
+  zero, unpriced yielding `?` for cloud and blank for local, projection
+  using the input rate.
+- Migration: an existing database gaining the three columns without loss,
+  and old rows reading as zero.
+- Failure isolation: an unreachable provider leaving local models listed.