path: root/docs/superpowers/specs/2026-08-09-external-providers-design.md

# External cloud providers

Date: 2026-08-09

## Goal

Let llamachat talk to OpenAI-compatible cloud providers (together.ai,
siliconflow) alongside the local llama.cpp router, without the local path
losing any of its current behaviour and without the existing config file
needing to be edited.

Cloud models cost money and carry no `presets.ini` entry, so two problems
come with them: the client has no context size or vision flag to work from,
and every turn resends the whole conversation at a real price. Both are
addressed by making the missing metadata enterable through a dialog and by
putting an approximate cost next to the context meter.

## Providers

`config.toml` grows a `[providers.*]` table. The local router stops being
special and becomes an ordinary entry named `default`.

```toml
[providers.default]
base_url = "http://localhost:8181"

[providers.together]
base_url = "https://api.together.xyz"
api_key = "pass:api/together"
filter = ["qwen", "deepseek"]
ctx_size = 32768
price_in = 0.60
price_out = 0.60
```

Per-provider keys, all optional except `base_url`:

| Key | Meaning |
| --- | --- |
| `base_url` | Required. API root, `/v1/...` is appended as today. |
| `api_key` | Absent means no `Authorization` header is sent. |
| `filter` | List of case-insensitive substrings. Empty or absent shows every model. |
| `ctx_size` | Prefill for the model dialog. Not applied to requests. |
| `vision` | Prefill for the model dialog. |
| `price_in` | USD per million input tokens. Prefill. |
| `price_out` | USD per million output tokens. Prefill. |

### Backward compatibility

A top-level `base_url` with no `[providers]` table synthesizes
`[providers.default]` from it. An existing config keeps working untouched
and nothing needs migrating. If both a top-level `base_url` and an explicit
`[providers.default]` are present, the explicit one wins.

### Model identity

Models are addressed as `provider:model`. The `default` provider is the
exception: its models are shown and stored bare, so the local dropdown reads
exactly as it does now and existing sessions keep resolving.

A model id containing a colon is therefore ambiguous only in principle; the
split is on the first colon, and any provider name that is not configured
falls back to treating the whole string as a `default` model. That keeps an
old session pointing at a local model working even if the model name itself
grows a colon.

### Filtering

Filtering applies to cloud providers so the dropdown stays usable:
together.ai lists roughly 200 models. A model is kept when any filter
substring appears in its id, compared case-insensitively. `"qwen"` matches
`Qwen/Qwen2.5-72B-Instruct-Turbo`.

The `default` provider is never filtered.

When a provider's filter matches nothing, the status bar reports it by name
with the counts, e.g. `together: 0 of 213 models matched filter`. The
provider contributes no models rather than silently falling back to all of
them.

A provider that cannot be reached does not break the model list: its models
are omitted and the failure is reported in the status bar. Local models must
remain selectable when the network is down.

## API keys

`api_key` is a single field, dispatched on prefix:

| Form | Behaviour |
| --- | --- |
| `pass:NAME` | Runs `pass show NAME`, takes the first line. |
| `env:VAR` | Reads the environment variable. |
| anything else | Used literally. |

Resolution is lazy: nothing runs until the first request to that provider,
so a session that only touches local models never invokes `pass` and never
triggers a pinentry. The resolved value is cached in memory for the process
lifetime and never written to disk.

The `pass` subprocess gets a 30 second timeout. A stuck pinentry then
surfaces as a backend error naming the provider instead of freezing the
worker thread. A missing entry, a non-zero exit, or empty output are all
reported the same way.

A literal key in `config.toml` works but the generated comments point at
`pass:` as the better option.

## Model metadata

Three layers resolve each of `ctx_size`, `vision`, `price_in`, `price_out`,
most specific first:

1. `models.ini` entry for that exact `provider:model`
2. `[providers.X]` default
3. unknown

Nothing has to be set in `config.toml`. The provider block is a convenience
that prefills the dialog; the dialog alone is enough.

Unknown degrades per value:

| Value | Unset behaviour |
| --- | --- |
| `ctx_size` | Meter shows tokens used with no limit and no percentage. Attachment budget uses a fallback. |
| `vision` | Treated as unknown, not as false: attaching an image is permitted and no dialog offers to switch models. |
| `price_in` / `price_out` | No cost figure. Shown as blank for `default`, `?` for a provider that has an `api_key`. |

The `?` distinguishes "local, free" from "cloud, price not entered". Both
would otherwise render as an empty label, which reads as free.

### models.ini

Written by the app, beside `config.toml` and `state.ini`, parsed with
configparser. Sections are keyed by full model id.

```ini
[together:Qwen/Qwen2.5-72B-Instruct-Turbo]
ctx_size = 32768
vision = false
price_in = 1.20
price_out = 1.20

[together:meta-llama/Llama-Vision-Free]
configured = false
```

`configured = false` records a cancelled dialog: the model was offered and
skipped, so it is never offered again. That is distinct from an absent
section, which means never asked.

The file grows one section per cloud model touched, including abandoned
ones. At the expected scale that is tens of entries in a plain text file the
user can prune by hand, so no cleanup logic is written.

### Model dialog

Four fields, prefilled from the provider defaults: context size, a vision
checkbox, input price, output price. Prices are per million tokens. Empty
price fields are valid and mean unpriced.

Two triggers:

- **First selection.** Choosing a model in the dropdown that has no
  `models.ini` section and belongs to a provider other than `default` opens
  the dialog once, before anything is typed. Cancel writes
  `configured = false`.
- **On demand.** A button beside the model dropdown opens the dialog for the
  current model, including local ones, and regardless of any previous
  cancel. This is how a wrong value gets corrected.

Local models are exempt from the automatic trigger because `presets.ini`
already answers context and vision for them and they cost nothing. The
on-demand path still reaches them, and a `models.ini` entry overrides the
preset when one exists.

## Cost

### Storage

`messages` gains three nullable columns:

| Column | Filled |
| --- | --- |
| `prompt_tokens` | On assistant rows, from the usage chunk. |
| `completion_tokens` | On assistant rows, from the usage chunk. |
| `model` | On assistant rows, the model that produced the reply. |

The migration is additive. Existing rows read as NULL and contribute zero,
so old conversations show a cost of zero rather than a fabricated one.

`model` is needed on the message rather than the session because `sessions`
records only the current model. A conversation that switched models would
otherwise be priced entirely at whichever model happens to be selected, and
switching mid-conversation is exactly what a mixed local-and-cloud setup
invites.

### Readout

Beside the context meter: the conversation's cost so far, plus what sending
the current draft would cost, shown separately so the two are not conflated:

```
$0.043 +$0.011
```

Cost so far is summed from the stored token counts, pricing each assistant
message by its own `model`, so it survives reopening a conversation. The
projected figure reuses the token estimate the context meter already
computes for the draft, priced at the currently selected model's input rate.

For an unpriced cloud model the figure is `?`. For a local model the label is
empty.

### Why the projection matters

`_chat_context` rebuilds every request from all stored rows, so reopening a
long conversation and sending one message resends the entire history. That
is free locally and billed per turn on a cloud provider. Showing the
projected cost of the next send is what makes that visible before it is
paid rather than after.

No trimming, summarization or context management is part of this work.
Visibility is the whole intervention: an expensive conversation is apparent,
and starting a new one is the user's call.

## Out of scope

- History trimming or summarization for long cloud conversations.
- A lifetime or per-month spend total. The stored counts make it a `SUM`
  away if it is ever wanted.
- Per-model tables in `config.toml`. The dialog covers that ground.
- Streaming or tool-call behaviour differences between providers, beyond not
  assuming they are absent. See the risk below.

## Risks

**Tool calling and reasoning content are unverified on cloud providers.**
The web search loop depends on tool-call fragments arriving in the delta's
`tool_calls` field and a `finish_reason` of `tool_calls`, and the reasoning
pane depends on `reasoning_content` deltas. Both are llama.cpp behaviours
that cloud providers implement with variations. Search on a cloud model may
need per-provider work once it is tried against a real endpoint. Nothing in
this design assumes it works out of the box, and local search behaviour must
not regress.

**A wrong price entered once is silently wrong forever.** The cost figure is
only as good as the numbers typed into the dialog, and no API publishes them
in a standard form. The readout is explicitly an approximation.

**Provider outages must not break the local path.** Model listing, key
resolution and cost display all have to fail soft, leaving local models
usable.

## Testing

Following the project's existing test approach, written before the
implementation:

- Config: providers parsed, bare `base_url` synthesizing `default`, explicit
  `default` winning over it, filter matching case-insensitively, empty
  filter meaning all, `default` never filtered.
- Key resolution: each prefix form, lazy resolution not firing for local
  requests, caching after first resolution, timeout and failure paths
  reported as errors.
- Model ids: split on first colon, unknown provider falling back to
  `default`, `default` models rendering bare.
- Metadata resolution: the three layers in order, unknown values degrading
  as tabulated, `configured = false` suppressing the dialog.
- `models.ini` round trip: written, reread, values preserved.
- Cost: summing per message by that message's model, NULL rows counting
  zero, unpriced yielding `?` for cloud and blank for local, projection
  using the input rate.
- Migration: an existing database gaining the three columns without loss,
  and old rows reading as zero.
- Failure isolation: an unreachable provider leaving local models listed.