cloud: the context size from the provider's /models

Hetzner runs vLLM, whose /models reports max_model_len per model
(262144); OpenRouter names it context_length. Read when the chat's model
changes, as /props is for a local one, so the meter shows a cloud chat's
fill and its attachment budget is half of what is free instead of a
fixed 24000 characters. Image input already went through: a cloud
model's inputs count as unknown.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

3 files changed, 80 insertions, 27 deletions

diff --git a/AGENTS.md b/AGENTS.md
index 645fc87..a9bd59f 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -59,6 +59,11 @@ socket (ask) ──> llama-server (LLM, tools) ──> socket (think, token, too
 - **Search** quotes the query as one FTS5 phrase with a prefix star. The
   self-check proves typed FTS syntax (`"`, `*`, `NOT`, `OR`, `)`) is
   matched as text and raises nothing.
+- **Usage** (2026-10-23): `stream()` keeps each request's last `usage`
+  chunk (Hetzner sends running totals on every chunk) and `answer()` saves
+  them per turn in `turns.usage`, JSON `{ts, provider, model, rounds}`,
+  `provider` the cloud name or `local`. Read by llm-cost-dashboard's
+  `ingest/assistant_ingest.py`. Titles and transcriptions are not counted.
 
 
 ## Hardware
@@ -284,7 +289,8 @@ and history were untouched.
 - `--cloud-models hetzner`: 4 ids in 1.2 s, including the pass prompt.
 - `models` listed the router's two and both configured cloud ones.
 - "Che tempo fa domani a Roma?": `weather` with `Rome,IT`, first token
-  4.6 s, ctx 775 then 953, `max` 0.
+  4.6 s, ctx 775 then 953, `max` 0 at the time. Hetzner's `/models`
+  reports `max_model_len` (262144); the meter reads it since.
 - "Quante email non lette ho?" in the same chat: no tool offered, it said
   it has no way to read mail. Nothing personal went out.
 - Voice clip (`05.wav`) in the cloud chat: transcribed by the local
diff --git a/assistant.py b/assistant.py
index 2bc91d5..173d767 100644
--- a/assistant.py
+++ b/assistant.py
@@ -189,10 +189,14 @@ def db_open(path):
     # for whatever the router has loaded.
     if "model" not in {r[1] for r in conn.execute("PRAGMA table_info(conversations)")}:
         conn.execute("ALTER TABLE conversations ADD COLUMN model TEXT")
+    # Added for llm-cost-dashboard: what the turn's requests cost in tokens,
+    # JSON {ts, provider, model, rounds: [the API's usage block per request]}.
+    if "usage" not in {r[1] for r in conn.execute("PRAGMA table_info(turns)")}:
+        conn.execute("ALTER TABLE turns ADD COLUMN usage TEXT")
     return conn
 
 
-def save_turn(conn, conv, user, reply, tools, messages, files=(), model=None):
+def save_turn(conn, conv, user, reply, tools, messages, files=(), model=None, usage=None):
     # Returns the conversation id: the row is made at its first finished turn,
     # so a New that is never used leaves nothing behind. A question that is
     # only files is titled by their names.
@@ -207,10 +211,10 @@ def save_turn(conn, conv, user, reply, tools, messages, files=(), model=None):
                          (now, model, conv))
         seq = conn.execute("SELECT COALESCE(MAX(seq), 0) + 1 FROM turns WHERE conv = ?",
                            (conv,)).fetchone()[0]
-        conn.execute("INSERT INTO turns(conv, seq, user, reply, tools, messages, files) "
-                     "VALUES (?, ?, ?, ?, ?, ?, ?)",
+        conn.execute("INSERT INTO turns(conv, seq, user, reply, tools, messages, files, usage) "
+                     "VALUES (?, ?, ?, ?, ?, ?, ?, ?)",
                      (conv, seq, user, reply, json.dumps(tools), json.dumps(messages),
-                      json.dumps(list(files))))
+                      json.dumps(list(files)), usage and json.dumps({"ts": now, **usage})))
     return conv
 
 
@@ -796,6 +800,32 @@ def route(model):
     return CFG["llm"], None, model, None
 
 
+def cloud_ctx(name, mid):
+    # A cloud model's context size, from its provider's /models: vLLM
+    # (Hetzner) reports max_model_len, OpenRouter context_length. 0 when it
+    # reports neither, and the meter shows tokens used only.
+    p = PROVIDERS[name]
+    req = urllib.request.Request(p["url"] + "/models",
+                                 headers={"Authorization": f"Bearer {p['key']}"})
+    try:
+        with urllib.request.urlopen(req, timeout=10) as r:
+            m = next((m for m in json.load(r)["data"] if m.get("id") == mid), {})
+        return int(m.get("max_model_len") or m.get("context_length") or 0)
+    except (OSError, KeyError, TypeError, ValueError):
+        return 0
+
+
+def ctx_for(model):
+    # The meter's model and size, asked again only when the model changes:
+    # /props for a local one, the provider's /models for a cloud one.
+    url, key, mid, cloud = route(model)
+    if not cloud:
+        return context_size(mid)
+    if ctx["model"] != model:
+        ctx.update(model=model, max=cloud_ctx(cloud, mid), used=0)
+    return ctx["max"]
+
+
 def cloud_models(name):
     # For the settings page (--cloud-models): the provider's model ids,
     # fetched here so the key never reaches it.
@@ -851,18 +881,18 @@ def llm_model():
     return model_inputs()[0]
 
 
-def ask(messages, calls, tools=None, mine=None, think=None, meter=True, model=None):
+def ask(messages, calls, tools=None, mine=None, think=None, meter=True, model=None, usage=None):
     # Yields ("think", text) and ("token", text) as they stream; tool-call
     # fragments go into calls. mine is the reply's turn: a stop shuts the
     # connection from the other thread (interrupt), which ends the reading.
     # model is the id model_inputs gave, None for the router's loaded one.
-    url, key, mid, cloud = route(model or llm_model())
+    # usage, a list, gets the request's usage block appended.
+    model = model or llm_model()
+    url, key, mid, cloud = route(model)
     # Only the chat's own replies move the meter: a title or a write still
     # running for another chat must not overwrite it.
-    if meter and cloud and ctx["model"] != model:
-        ctx.update(model=model, max=0, used=0)   # no /props: tokens used only
-    elif meter and not cloud:
-        context_size(mid)
+    if meter:
+        ctx_for(model)
     body = {"model": mid, "messages": messages, "stream": True,
             "stream_options": {"include_usage": True}}
     if not cloud:
@@ -882,7 +912,7 @@ def ask(messages, calls, tools=None, mine=None, think=None, meter=True, model=No
             return                        # stopped before the request went out
         inflight["conn"] = conn
     try:
-        yield from stream(conn, u.path, body, calls, meter, key, cloud or "llama-server")
+        yield from stream(conn, u.path, body, calls, meter, key, cloud or "llama-server", usage)
     except OSError:
         if mine is not None and turn != mine:
             return                        # the stop hung up under the read
@@ -894,7 +924,7 @@ def ask(messages, calls, tools=None, mine=None, think=None, meter=True, model=No
         conn.close()
 
 
-def stream(conn, path, body, calls, meter=True, key=None, who="llama-server"):
+def stream(conn, path, body, calls, meter=True, key=None, who="llama-server", usage=None):
     headers = {"Content-Type": "application/json"}
     if key:
         headers["Authorization"] = f"Bearer {key}"
@@ -908,7 +938,7 @@ def stream(conn, path, body, calls, meter=True, key=None, who="llama-server"):
             why = why.replace(key, "***")   # some providers echo the key back
         raise RuntimeError(f"{who} {r.status}: {why[:2000]}")
     with r:
-        streamed, other, metered = False, b"", False
+        streamed, other, metered, last = False, b"", False, None
         for line in r:
             if not line.startswith(b"data: "):
                 # The router answers a crashed model with a plain line, e.g.
@@ -922,6 +952,9 @@ def stream(conn, path, body, calls, meter=True, key=None, who="llama-server"):
             if "error" in chunk:          # a failure mid-stream arrives as a chunk
                 err = chunk["error"]
                 raise RuntimeError(err.get("message", str(err)) if isinstance(err, dict) else str(err))
+            # The last usage counts: a provider sending it with every chunk
+            # sends running totals.
+            last = chunk.get("usage") or last
             # Once per request: llama-server sends usage at the end, some
             # providers (Hetzner) with every chunk.
             if chunk.get("usage") and meter and not metered:
@@ -945,6 +978,8 @@ def stream(conn, path, body, calls, meter=True, key=None, who="llama-server"):
                 yield "token", delta["content"]
         if not streamed:
             raise RuntimeError(other.decode(errors="replace")[:300] or f"{who} sent nothing")
+        if usage is not None and last:
+            usage.append(last)
 
 
 def say(mine, kind, value, done=False):
@@ -1112,13 +1147,14 @@ def reply(mine, text, files=(), audio=None, model=None):
 
 def answer(mine, text, files=(), chosen=None):
     blocks, refs, chips = [], [], []
-    if files:
-        # Half of what the context has free, at about 3 characters a token.
-        free = (ctx["max"] - ctx["used"]) // 2 * 3 if ctx["max"] else 24000
-        blocks, refs, chips = attach(files, max(free, 2000))
     # A server that does not report inputs is left to decide for itself.
     model, inputs = model_inputs(chosen)
     images = inputs is None or "image" in inputs
+    if files:
+        # Half of what this chat's model has free, at about 3 characters a token.
+        ctx_for(model)
+        free = (ctx["max"] - ctx["used"]) // 2 * 3 if ctx["max"] else 24000
+        blocks, refs, chips = attach(files, max(free, 2000))
     # A cloud chat is offered only the tools marked cloud = true: the others
     # would send mail, money or calendar to the provider. One of them called
     # anyway (copied from history) answers "no tool named".
@@ -1153,14 +1189,14 @@ def answer(mine, text, files=(), chosen=None):
     # tool round would cost seconds between rounds, and a file changed
     # mid-answer would flip from image to gone.
     sent = wire(msgs, images)
-    said, used = "", []
+    said, used, rounds = "", [], []
     # A stop shuts this reply's connection (interrupt), so llama-server's slot
     # frees even mid prompt pass. ponytail: a running tool still runs out.
     for rnd in range(CFG["tool_rounds"] + 1):
         calls, content = {}, ""
         # The last round offers no tools, which forces an answer.
         for kind, piece in ask(sent, calls, offer if rnd < CFG["tool_rounds"] else None,
-                               mine, model=model):
+                               mine, model=model, usage=rounds):
             if not say(mine, kind, piece):
                 log("interrupted")
                 return
@@ -1202,7 +1238,10 @@ def answer(mine, text, files=(), chosen=None):
             log("interrupted")
             return
         fresh = cur["id"] is None
-        cur["id"] = save_turn(DB, cur["id"], text, said, used, entry, chips, chosen)
+        _, _, mid, cloud = route(model)
+        # ponytail: titles and transcriptions are not counted, always local
+        cur["id"] = save_turn(DB, cur["id"], text, said, used, entry, chips, chosen,
+                              {"provider": cloud or "local", "model": mid, "rounds": rounds})
         conv = cur["id"]
         cur["history"].append(entry)
         cur["turns"].append({"user": text, "reply": said, "tools": used, "files": chips})
@@ -1518,8 +1557,10 @@ def test():
         def do_GET(self):
             if self.headers.get("Authorization"):      # cloud_models; the router's has none
                 script["auth"] = self.headers["Authorization"]
-            body = ({"data": [{"id": "m", "status": {"value": "loaded"},
-                               "architecture": {"input_modalities": script["inputs"]}}]}
+            body = ({"data": [{"id": "vendor/x:free", "max_model_len": 65536}, {"id": "other"}]}
+                    if self.path.startswith("/cloud/") and self.path.endswith("/models")
+                    else {"data": [{"id": "m", "status": {"value": "loaded"},
+                                    "architecture": {"input_modalities": script["inputs"]}}]}
                     if self.path.endswith("/models")
                     else {"default_generation_settings": {"n_ctx": 4096}})
             self.send_response(200)
@@ -1594,7 +1635,7 @@ def test():
     more = [{"name": n, "url": base, "key_cmd": c, "models": []}
             for n, c in (("pass", "printf 'sk-x\\nlogin: y\\n'"), ("spaced", "printf 'a b'"))]
     assert {n: p["key"] for n, p in load_providers(more).items()} == {"pass": "sk-x"}
-    assert cloud_models("fake") == ["m"] and script["auth"] == "Bearer sk-test"
+    assert cloud_models("fake") == ["other", "vendor/x:free"] and script["auth"] == "Bearer sk-test"
     for name, why in (("broken", "key_cmd failed"), ("nope", "no cloud provider")):
         try:
             cloud_models(name)
@@ -1892,7 +1933,7 @@ def test():
                                               "content": "error: no tool named mail"}
             assert ("think", "hm") in [(e["type"], e.get("delta")) for e in seen]
             meters = [e for e in seen if e["type"] == "ctx"]
-            assert meters[-1]["model"] == "fake:vendor/x:free" and meters[-1]["max"] == 0
+            assert meters[-1]["model"] == "fake:vendor/x:free" and meters[-1]["max"] == 65536
             assert "sk-test" not in json.dumps(seen)
             conv = cur["id"]
             assert DB.execute("SELECT model FROM conversations WHERE id = ?",
@@ -1904,6 +1945,9 @@ def test():
             send(type="ask", text="again")
             meters = [e for e in until("done") if e["type"] == "ctx"]
             assert [e["used"] for e in meters] == [7], meters
+            u = json.loads(DB.execute("SELECT usage FROM turns ORDER BY id DESC").fetchone()[0])
+            assert (u["provider"], u["model"], u["rounds"]) == (
+                "fake", "vendor/x:free", [{"prompt_tokens": 42}]) and u["ts"], u   # the last block
             assert script["paths"][-1] == "/cloud/v1/chat/completions"
             # Reopened, the chat reports its model.
             send(type="new")
diff --git a/docs/superpowers/specs/2026-10-23-cloud-models-design.md b/docs/superpowers/specs/2026-10-23-cloud-models-design.md
index 8a581ee..d2270af 100644
--- a/docs/superpowers/specs/2026-10-23-cloud-models-design.md
+++ b/docs/superpowers/specs/2026-10-23-cloud-models-design.md
@@ -69,8 +69,11 @@ ids contain `/` and sometimes `:` (OpenRouter's `:free`).
   asks for a model" becomes "only for a model the user picked for this
   chat". `ctx` still carries the model actually answering.
 - **Cloud model.** Same request body minus `chat_template_kwargs` (strict
-  providers reject unknown fields). No `/props`: `ctx` max is 0 and the
-  canvas shows used tokens only, from `usage`. Reasoning deltas are read
+  providers reject unknown fields). No `/props`: `ctx` max comes from the
+  provider's `/models`, `max_model_len` (vLLM, Hetzner) or
+  `context_length` (OpenRouter), read when the chat's model changes; 0
+  when neither is there, and the canvas then shows used tokens only. A
+  chat's attachment budget uses the same size. Reasoning deltas are read
   from `reasoning_content` (llama-server) or `reasoning` (OpenRouter).
   `temperature` from config applies when set. Vision and audio are
   unknown for cloud models (`/models` does not report them), so