aboutsummaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authorDanilo M. <danix@danix.xyz>2026-08-21 20:58:39 +0200
committerDanilo M. <danix@danix.xyz>2026-08-21 20:58:39 +0200
commita7422ddeb7771983e984350b31092fe4898897c6 (patch)
tree2833a9c8f5c2ab646ceee5f42b711b7d0c35ec47
parent2cae7f19ac90e6df9d4008e6e364bae30e8b389a (diff)
downloadllamachat-feature/external-providers.tar.gz
llamachat-feature/external-providers.zip
fix: web search on DeepSeek and other reasoning modelsfeature/external-providers
A searched turn on a cloud reasoning model ended at the thinking: the model emitted the tool call, but finish_reason: "tool_calls" landed on the same SSE line as the include_usage block, so the single-event parser returned that line as a usage chunk and the loop never saw the tool finish. The parser now emits every event a line carries, so the search fires. Also in this change: - Replay each round's reasoning_content on the assistant tool-call message, which interleaved-thinking models require to keep going. - Add a per-provider replay_reasoning option (DeepSeek, SiliconFlow GLM-4.7+) to carry prior turns' reasoning_content when search is on. - Add an on-demand diagnostic log gated by $LLAMACHAT_DEBUG_LOG. - Record provider, usage_json and reported_cost_usd per reply, so a searched turn keeps every round's billed usage for external consumers.
-rw-r--r--CHANGELOG.md27
-rw-r--r--README.md28
-rw-r--r--llamachat/backend.py118
-rw-r--r--llamachat/config.py8
-rw-r--r--llamachat/db.py24
-rw-r--r--llamachat/providers.py22
-rw-r--r--llamachat/ui.py76
-rwxr-xr-xtest_llamachat.py328
8 files changed, 584 insertions, 47 deletions
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 6e9a831..34314f8 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -34,10 +34,28 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
`model`. Token counts come from the stream's final usage chunk and power the
cost readout; the model column records which model actually replied, so a
switched conversation prices each turn correctly.
+- Three more columns on `messages`: `provider`, `usage_json` and
+ `reported_cost_usd`. The provider name is recorded when a reply finishes so
+ history stays self-describing even if a provider is later renamed or
+ removed; `usage_json` keeps every round's raw usage block from the stream
+ as a JSON array (a searched turn makes one API call per search round, and
+ all of them are billed, so all of them survive) for external consumers
+ such as the cost dashboard's ingester; and `reported_cost_usd` stores the
+ GUI's hand-entered-price estimate for the reply, summed across rounds, as
+ a sanity check only, staying NULL for an unpriced model rather than
+ claiming it cost nothing.
- Per-provider `thinking_budget` option, for endpoints (currently SiliconFlow)
that cap chain-of-thought tokens separately from the final answer. Unset
providers skip the key entirely so other endpoints do not receive an unknown
parameter.
+- Per-provider `replay_reasoning` option. When set and web search is enabled,
+ each prior assistant turn's `reasoning_content` is replayed on the next
+ request, which interleaved-thinking providers (DeepSeek, SiliconFlow's
+ GLM-4.7+) require when a `tools` key is present. Off by default so other
+ providers do not pay context and input tokens for thinking they ignore.
+- On-demand diagnostic log. Setting `LLAMACHAT_DEBUG_LOG` to a path appends
+ timestamped provider-error records (status, request body, full response) to
+ that file; unset, it writes nothing, so a normal launch never grows a log.
### Fixed
@@ -48,6 +66,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
which could be too small for a reasoning model. Non-local providers now send
`max_tokens = 32768`, leaving room for both chain-of-thought reasoning and
the answer.
+- A streamed tool call is now detected even when the provider puts
+ `finish_reason: "tool_calls"` on the same SSE line as the usage block (the
+ `include_usage` final chunk). The parser previously returned that line as
+ `usage` and dropped the finish signal, so a DeepSeek web search never
+ triggered and the turn ended at the thinking.
+- A searched turn now replays the model's `reasoning_content` verbatim on the
+ assistant tool-call message. Interleaved-thinking models (DeepSeek V3.2+/V4,
+ GLM-4.7+ on SiliconFlow) require this and stop answering — ending the turn at
+ the thinking — when it is dropped.
Tool calling and reasoning output vary between providers. Web search on a
cloud model may not work as reliably as it does with a local router.
diff --git a/README.md b/README.md
index 5df70d6..416a84c 100644
--- a/README.md
+++ b/README.md
@@ -288,7 +288,9 @@ there is no second thread.
```sql
sessions (id, mode, title, model, prompt_name, prompt_custom,
created_at, updated_at)
-messages (id, session_id, role, content, reasoning, searches, created_at)
+messages (id, session_id, role, content, reasoning, searches,
+ prompt_tokens, completion_tokens, model, provider, usage_json,
+ reported_cost_usd, created_at)
attachments (id, message_id, path, kind, mime, size, sha256, thumb, truncated)
messages_fts -- FTS5 external-content table over messages.content
```
@@ -307,6 +309,19 @@ per search with its query, results and any error, or NULL when nothing was
searched. Like `reasoning` it stays out of the FTS index, or snippets from
web pages would compete with messages the user actually wrote.
+`provider` records which provider served the reply (`local` or a configured
+provider name), written when the reply finishes so history stays
+self-describing even if `config.toml` later renames or drops the provider.
+`usage_json` keeps every round's raw usage block from the stream as a JSON
+array — a searched turn makes one API call per search round, and each round
+is billed separately, so all of them are kept verbatim and token details
+beyond the two counts (e.g. cached or reasoning tokens on a cloud endpoint)
+are recoverable later without re-guessing a field mapping.
+`reported_cost_usd` is the GUI's hand-entered-price estimate for that one
+reply (summed across rounds), stored as a sanity check for external
+consumers — it is approximate and never a source of truth; it stays NULL for
+a model with no entered prices rather than claiming the reply cost nothing.
+
`prompt_name` records which system prompt a conversation was built with, and
`prompt_custom` holds the text when that prompt is a one-off rather than a
file. Storing the name rather than the resolved text means editing a preset
@@ -463,6 +478,8 @@ filter = ["qwen", "deepseek"]
ctx_size = 32768
price_in = 0.60
price_out = 0.60
+# thinking_budget = 8192 # per-provider chain-of-thought cap
+# replay_reasoning = false # see below
```
`api_key` accepts three forms:
@@ -499,6 +516,15 @@ local router, but on a cloud provider every message is billed for the entire
conversation so far. The projection in the cost label exists to make that
visible before you send.
+`thinking_budget` sets a provider-specific cap on chain-of-thought tokens
+(currently sent to SiliconFlow; other endpoints ignore it when unset).
+
+`replay_reasoning` makes each assistant turn's thinking be sent back on the
+next request while web search is enabled. DeepSeek's and SiliconFlow's
+interleaved-thinking models require this when a `tools` key is present, and
+return a 400 if it is missing. Leave it off elsewhere — replaying thinking
+costs context and input tokens for providers that ignore it.
+
Tool calling and reasoning output vary between providers. Web search on a
cloud model may not work as reliably as it does with a local router.
diff --git a/llamachat/backend.py b/llamachat/backend.py
index 4f15936..338f396 100644
--- a/llamachat/backend.py
+++ b/llamachat/backend.py
@@ -18,7 +18,9 @@ import hashlib
import io
import json
import mimetypes
+import os
from dataclasses import dataclass, field
+from datetime import datetime
from pathlib import Path
import httpx
@@ -46,6 +48,26 @@ IMAGE_MIMES = {"image/png", "image/jpeg", "image/gif", "image/webp", "image/bmp"
DONE = ("done", "")
+def _debug_log(entry: str) -> None:
+ """Append one diagnostic record to $LLAMACHAT_DEBUG_LOG, when set.
+
+ On-demand by design: unset (the default) this is a no-op and no file is
+ ever created, so a normal launch never grows a log to rotate. Set the
+ variable to a path to capture provider errors while chasing one, then
+ unset it again. The request body is safe to log: the API key travels in
+ the Authorization header, never in the JSON body.
+ """
+ path = os.environ.get("LLAMACHAT_DEBUG_LOG")
+ if not path:
+ return
+ stamp = datetime.now().isoformat(timespec="seconds")
+ try:
+ with open(path, "a", encoding="utf-8") as fh:
+ fh.write(f"[{stamp}] {entry}\n")
+ except OSError:
+ pass # a bad log path must never take down the chat
+
+
class BackendError(Exception):
"""Any failure talking to the router, already phrased for the user."""
@@ -206,8 +228,10 @@ class Client:
resp.raise_for_status()
payload = resp.json()
except httpx.HTTPError as exc:
+ _debug_log(f"GET /v1/models failed: {exc}")
raise BackendError(f"Cannot reach router at {self.base_url}: {exc}")
except ValueError as exc:
+ _debug_log(f"GET /v1/models sent invalid JSON: {exc}")
raise BackendError(f"Router sent invalid JSON: {exc}")
# Some OpenAI-compatible providers return a bare array instead of
# the standard {"data": [...]} envelope. Accept both.
@@ -241,8 +265,10 @@ class Client:
resp.raise_for_status()
payload = resp.json()
except httpx.HTTPError as exc:
+ _debug_log(f"POST /v1/chat/completions (complete) failed: {exc}")
raise BackendError(f"Request failed: {exc}")
except ValueError as exc:
+ _debug_log(f"POST /v1/chat/completions (complete) invalid JSON: {exc}")
raise BackendError(f"Router sent invalid JSON: {exc}")
choices = payload.get("choices") or []
if not choices:
@@ -280,6 +306,7 @@ class Client:
tools = None if last_round else [search.TOOL_SCHEMA]
calls: dict = {}
saw_tool_finish = False
+ reasoning = ""
for kind, piece in self._stream_once(model, convo, tools=tools):
if kind == "tool_calls":
@@ -287,6 +314,8 @@ class Client:
elif kind == "tool_finish":
saw_tool_finish = True
else:
+ if kind == "reasoning":
+ reasoning += piece
yield (kind, piece)
wanted = [
@@ -297,23 +326,30 @@ class Client:
if not (saw_tool_finish and wanted) or last_round:
return
- convo.append(
- {
- "role": "assistant",
- "content": None,
- "tool_calls": [
- {
- "id": call["id"],
- "type": "function",
- "function": {
- "name": call["name"],
- "arguments": call["arguments"],
- },
- }
- for call in wanted
- ],
- }
- )
+ assistant = {
+ "role": "assistant",
+ "content": None,
+ "tool_calls": [
+ {
+ "id": call["id"],
+ "type": "function",
+ "function": {
+ "name": call["name"],
+ "arguments": call["arguments"],
+ },
+ }
+ for call in wanted
+ ],
+ }
+ # Interleaved-thinking models (DeepSeek V3.2+/V4, GLM-4.7+ on
+ # SiliconFlow) emit their chain-of-thought as reasoning_content and
+ # require it replayed verbatim on the assistant tool-call message.
+ # Dropping it breaks their multi-step tool flow: the follow-up
+ # round ends with the model thinking and no answer. Local reasoning
+ # models tolerate the extra key, so include it whenever it exists.
+ if reasoning:
+ assistant["reasoning_content"] = reasoning
+ convo.append(assistant)
# The round after this one is the last, so its results are the
# last the model will get. Saying so is what makes it answer
# instead of asking for a search it can no longer make.
@@ -392,17 +428,21 @@ class Client:
) as resp:
if resp.status_code != 200:
resp.read()
+ _debug_log(
+ f"POST /v1/chat/completions -> {resp.status_code}\n"
+ f"request: {json.dumps(body, ensure_ascii=False)}\n"
+ f"response: {resp.text}"
+ )
raise BackendError(
f"Router returned {resp.status_code}: {resp.text[:300]}"
)
for line in resp.iter_lines():
chunk = _parse_sse_line(line)
- if chunk is None:
- continue
if chunk == DONE:
return
- yield chunk
+ yield from chunk
except httpx.HTTPError as exc:
+ _debug_log(f"POST /v1/chat/completions transport error: {exc}")
raise BackendError(f"Request failed: {exc}")
@@ -588,46 +628,54 @@ def estimate_tokens(messages: list[dict], chars_per_token: float) -> int:
return int(chars / max(chars_per_token, 1.0)) + overhead + images * 600
-def _parse_sse_line(line: str) -> tuple[str, str] | None:
- """Decode one SSE line.
+def _parse_sse_line(line: str):
+ """Decode one SSE line into zero or more (kind, text) events.
- Returns a ('reasoning'|'content', text) pair, the DONE sentinel at the
- end of the stream, or None for lines carrying nothing to render.
+ Returns a list, empty for lines carrying nothing to render, or the DONE
+ sentinel at the end of the stream. One line can carry two events: with
+ `stream_options.include_usage` the final chunk holds both the
+ finish_reason and the usage block together, and a searched turn needs
+ both — the former marks the tool call, the latter its token counts.
"""
if not line.startswith("data:"):
- return None
+ return []
payload = line[5:].strip()
if payload == "[DONE]":
return DONE
try:
obj = json.loads(payload)
except ValueError:
- return None
+ return []
+
+ events: list[tuple[str, str]] = []
+
# The usage chunk arrives with an empty choices list, so check it first.
usage = obj.get("usage")
if usage and usage.get("prompt_tokens") is not None:
- return ("usage", json.dumps(usage))
+ events.append(("usage", json.dumps(usage)))
choices = obj.get("choices") or []
if not choices:
- return None
+ return events
delta = choices[0].get("delta") or {}
# Tool-call fragments arrive in their own delta field, alongside a
# finish_reason of 'tool_calls' on the final chunk of the round.
tool_calls = delta.get("tool_calls")
if tool_calls:
- return ("tool_calls", json.dumps(tool_calls))
+ events.append(("tool_calls", json.dumps(tool_calls)))
if choices[0].get("finish_reason") == "tool_calls":
- return ("tool_finish", "")
+ events.append(("tool_finish", ""))
# Both fields can arrive in one delta, and a reasoning delta can be an
# empty string. Test for presence, not truthiness, so a chunk carrying
# reasoning is never mistaken for a content chunk.
reasoning = delta.get("reasoning_content")
if reasoning is not None:
- return ("reasoning", reasoning) if reasoning else None
- content = delta.get("content")
- if content:
- return ("content", content)
- return None
+ if reasoning:
+ events.append(("reasoning", reasoning))
+ else:
+ content = delta.get("content")
+ if content:
+ events.append(("content", content))
+ return events
diff --git a/llamachat/config.py b/llamachat/config.py
index 81b799b..0c119ad 100644
--- a/llamachat/config.py
+++ b/llamachat/config.py
@@ -253,6 +253,13 @@ def write_default(path: Path = CONFIG_PATH) -> Path:
'# default is 4096; raise it for complex questions that need more\n'
'# room to reason.\n'
'#\n'
+ '# replay_reasoning makes each assistant turn\'s thinking be sent back\n'
+ '# on the next request while web search is enabled. DeepSeek and\n'
+ '# SiliconFlow\'s interleaved-thinking models reject a tools request\n'
+ '# whose prior assistant turns omit reasoning_content, so set it for\n'
+ '# those; leave it off elsewhere, since replaying thinking costs\n'
+ '# context and input tokens.\n'
+ '#\n'
'# [providers.together]\n'
'# base_url = "https://api.together.xyz"\n'
'# api_key = "pass:api/together"\n'
@@ -261,6 +268,7 @@ def write_default(path: Path = CONFIG_PATH) -> Path:
'# price_in = 0.60\n'
'# price_out = 0.60\n'
'# thinking_budget = 8192\n'
+ '# replay_reasoning = false\n'
)
return path
diff --git a/llamachat/db.py b/llamachat/db.py
index e88c131..9ec220a 100644
--- a/llamachat/db.py
+++ b/llamachat/db.py
@@ -42,6 +42,9 @@ CREATE TABLE IF NOT EXISTS messages (
prompt_tokens INTEGER,
completion_tokens INTEGER,
model TEXT,
+ provider TEXT, -- provider name at write time ('local' or a configured one)
+ usage_json TEXT, -- raw usage block from the stream's final SSE chunk
+ reported_cost_usd REAL, -- GUI's hand-entered-price estimate; sanity check only
created_at INTEGER NOT NULL
);
@@ -117,6 +120,21 @@ class History:
# The model on `sessions` is the current one, which prices a
# switched conversation wrongly. Record what actually replied.
"model": "TEXT",
+ # Nullable: pre-migration rows have no provider recorded, and
+ # the ingester's ':'-split fallback covers those. Storing it
+ # at write time beats re-deriving it from config.toml later,
+ # which the user may rename or remove.
+ "provider": "TEXT",
+ # The raw usage block from the stream's final SSE chunk, kept
+ # verbatim like the dashboard's raw_json: a future field
+ # mapping (e.g. cached tokens) is re-derivable, not guessed.
+ "usage_json": "TEXT",
+ # The GUI's hand-entered-price estimate for this reply. A
+ # sanity check only -- real numbers come from the dashboard's
+ # pricing engine. NULL (unpriced model) is never written as
+ # 0.0: zero would claim a reply cost nothing when the GUI
+ # simply could not say.
+ "reported_cost_usd": "REAL",
},
"sessions": {
"prompt_name": "TEXT NOT NULL DEFAULT ''",
@@ -226,6 +244,9 @@ class History:
prompt_tokens: int | None = None,
completion_tokens: int | None = None,
model: str | None = None,
+ provider: str | None = None,
+ usage_json: str | None = None,
+ reported_cost_usd: float | None = None,
) -> None:
"""Fill in a streamed reply. Omitted fields keep their stored value.
@@ -242,6 +263,9 @@ class History:
("prompt_tokens", prompt_tokens),
("completion_tokens", completion_tokens),
("model", model),
+ ("provider", provider),
+ ("usage_json", usage_json),
+ ("reported_cost_usd", reported_cost_usd),
):
if value is not None:
columns.append(f"{column} = ?")
diff --git a/llamachat/providers.py b/llamachat/providers.py
index a8b2350..23ad081 100644
--- a/llamachat/providers.py
+++ b/llamachat/providers.py
@@ -49,6 +49,12 @@ class Provider:
# Provider-specific reasoning token budget. Currently SiliconFlow only;
# ignored when unset so other endpoints do not receive an unknown key.
thinking_budget: int | None = None
+ # Replay each assistant turn's reasoning_content on the next request.
+ # Interleaved-thinking providers (DeepSeek, SiliconFlow's GLM-4.7+) reject
+ # a tools request whose prior assistant turns omit it. Off by default:
+ # replaying thinking inflates context and the input bill for providers
+ # that neither need nor want it.
+ replay_reasoning: bool = False
@property
def is_local(self) -> bool:
@@ -189,6 +195,7 @@ def parse(values: dict, warnings: list[str] | None = None) -> dict[str, Provider
price_in=_number(entry.get("price_in"), float),
price_out=_number(entry.get("price_out"), float),
thinking_budget=_number(entry.get("thinking_budget"), int),
+ replay_reasoning=bool(entry.get("replay_reasoning", False)),
)
return out
@@ -220,6 +227,21 @@ def split(model_id: str, table: dict[str, Provider]) -> tuple[str, str]:
return LOCAL, model_id
+def replays_reasoning(model_id: str, table: dict[str, Provider], search_enabled: bool) -> bool:
+ """Whether a request to this model must replay reasoning_content.
+
+ Only true when search is enabled, because that is the only time the app
+ sends a `tools` key — the condition under which interleaved-thinking
+ providers demand the reasoning back. The caller still checks that a given
+ row has non-empty reasoning; a non-reasoning model has nothing to replay.
+ """
+ if not search_enabled:
+ return False
+ name, _ = split(model_id, table)
+ provider = table.get(name)
+ return bool(provider and provider.replay_reasoning)
+
+
def apply_filter(provider: Provider, listed: list[str]) -> list[str]:
"""Keep models matching any of the provider's substrings.
diff --git a/llamachat/ui.py b/llamachat/ui.py
index 189d59a..32f7242 100644
--- a/llamachat/ui.py
+++ b/llamachat/ui.py
@@ -193,6 +193,7 @@ class StreamWorker(QObject):
chunk = Signal(str)
reasoning = Signal(str)
usage = Signal(int, int)
+ usage_all = Signal(str) # every round's usage block, as a JSON array
search_start = Signal(str)
search_done = Signal(str)
finished = Signal()
@@ -211,6 +212,7 @@ class StreamWorker(QObject):
self.messages = messages
self.search_cfg = search_cfg
self._stop = False
+ self._usage_blocks: list[str] = []
def stop(self) -> None:
self._stop = True
@@ -243,6 +245,11 @@ class StreamWorker(QObject):
self.search_done.emit(piece)
elif kind == "usage":
stats = json.loads(piece)
+ # A searched turn makes several API calls (one per search
+ # round); every round's usage block must survive for the
+ # dashboard to price each billed request, not just the
+ # last one the meter shows.
+ self._usage_blocks.append(piece)
self.usage.emit(
int(stats.get("prompt_tokens") or 0),
int(stats.get("total_tokens") or 0),
@@ -255,6 +262,9 @@ class StreamWorker(QObject):
except Exception as exc: # noqa: BLE001 - surface anything as UI text
self.failed.emit(f"Unexpected error: {exc}")
return
+ self.usage_all.emit(
+ json.dumps([json.loads(b) for b in self._usage_blocks])
+ )
self.finished.emit()
@@ -461,6 +471,7 @@ class ChatWindow(QMainWindow):
self.searches: list[dict] = []
self.turn_prompt_tokens: int | None = None
self.turn_completion_tokens: int | None = None
+ self.turn_usage_json: str | None = None
self._conversation_cost = 0.0
# Every rendered bubble, so a reasoning toggle can redraw the
# transcript without refetching anything.
@@ -1127,8 +1138,18 @@ class ChatWindow(QMainWindow):
"""Prior turns plus the new message, for multi-turn mode."""
messages = self._system_messages()
rows = self.history.messages(self.session_id)
+ # Interleaved-thinking providers require each assistant turn's
+ # reasoning_content back on the next tools request (see
+ # providers.replays_reasoning). Non-empty only, so a plain model
+ # never carries an empty key it never produced.
+ replay = providers_mod.replays_reasoning(
+ self.current_model(), self.cfg.providers, self.cfg.search_enabled
+ )
for row in rows[:-1]: # the latest user row is replaced below
- messages.append({"role": row["role"], "content": row["content"]})
+ message = {"role": row["role"], "content": row["content"]}
+ if replay and row["role"] == "assistant" and row["reasoning"]:
+ message["reasoning_content"] = row["reasoning"]
+ messages.append(message)
messages.append({"role": "user", "content": latest_content})
return messages
@@ -1240,6 +1261,7 @@ class ChatWindow(QMainWindow):
self.worker.search_start.connect(self._on_search_start)
self.worker.search_done.connect(self._on_search_done)
self.worker.usage.connect(self._on_usage)
+ self.worker.usage_all.connect(self._on_usage_all)
self.worker.finished.connect(self._on_stream_finished)
self.worker.failed.connect(self._on_stream_failed)
self.thread.start()
@@ -1264,6 +1286,18 @@ class ChatWindow(QMainWindow):
self.update_cost()
@Slot(str)
+ def _on_usage_all(self, blocks: str) -> None:
+ """Keep every round's usage block for the finished turn.
+
+ A searched turn makes several API calls (one per search round) and
+ the dashboard prices each round's tokens, so all of them must
+ survive -- not just the last one the meter shows. The block is a
+ JSON array of the raw per-round usage objects; the ingester sums
+ what it needs.
+ """
+ self.turn_usage_json = blocks if blocks != "[]" else None
+
+ @Slot(str)
def _on_reasoning(self, piece: str) -> None:
if not self.reasoning_buffer:
self.hide_status()
@@ -1289,6 +1323,42 @@ class ChatWindow(QMainWindow):
"""Stored form of this turn's searches: NULL when there were none."""
return json.dumps(self.searches) if self.searches else None
+ def _current_provider(self) -> str:
+ """The provider that served the current model, as stored.
+
+ 'local' for a bare model id, the configured provider name otherwise
+ (providers.split resolves against the provider table). Stored per
+ reply so history stays self-describing even if config.toml later
+ renames or drops the provider.
+ """
+ return providers_mod.split(self.current_model(), self.cfg.providers)[0]
+
+ def _turn_reported_cost(self) -> float | None:
+ """The GUI's hand-entered-price estimate for the finished reply.
+
+ NULL for a model with no entered prices (is_priced): 0.0 would
+ claim the reply cost nothing when the GUI simply cannot say. The
+ dashboard treats this as a sanity check only, never as a source of
+ truth. Sums every round of a searched turn: each round is its own
+ billed request.
+ """
+ model_id = self.current_model()
+ if not models_mod.is_priced(model_id, self.cfg.providers, self.store):
+ return None
+ prompt = 0
+ completion = 0
+ for block in json.loads(self.turn_usage_json or "[]"):
+ prompt += block.get("prompt_tokens") or 0
+ completion += block.get("completion_tokens") or 0
+ if not (prompt or completion):
+ return None
+ row = {
+ "model": model_id,
+ "prompt_tokens": prompt,
+ "completion_tokens": completion,
+ }
+ return models_mod.message_cost(row, self.cfg.providers, self.store)
+
@Slot()
def _on_stream_finished(self) -> None:
if self.assistant_message_id is not None:
@@ -1300,6 +1370,9 @@ class ChatWindow(QMainWindow):
prompt_tokens=self.turn_prompt_tokens,
completion_tokens=self.turn_completion_tokens,
model=self.current_model(),
+ provider=self._current_provider(),
+ usage_json=self.turn_usage_json,
+ reported_cost_usd=self._turn_reported_cost(),
)
rows = self.history.messages(self.session_id)
self._conversation_cost = models_mod.conversation_cost(
@@ -1382,6 +1455,7 @@ class ChatWindow(QMainWindow):
self.assistant_message_id = None
self.turn_prompt_tokens = None
self.turn_completion_tokens = None
+ self.turn_usage_json = None
self.send_button.setEnabled(True)
self.stop_button.hide()
diff --git a/test_llamachat.py b/test_llamachat.py
index 3f1f678..74b0d0c 100755
--- a/test_llamachat.py
+++ b/test_llamachat.py
@@ -208,24 +208,24 @@ def test_classify():
def test_sse_parsing():
line = 'data: {"choices":[{"delta":{"content":"hi"}}]}'
- assert backend._parse_sse_line(line) == ("content", "hi")
+ assert backend._parse_sse_line(line) == [("content", "hi")]
# Reasoning arrives in its own field, which is what lets the UI keep
# thinking and reply apart without parsing <think> tags.
think = 'data: {"choices":[{"delta":{"reasoning_content":"hmm"}}]}'
- assert backend._parse_sse_line(think) == ("reasoning", "hmm")
+ assert backend._parse_sse_line(think) == [("reasoning", "hmm")]
assert backend._parse_sse_line("data: [DONE]") == backend.DONE
- assert backend._parse_sse_line(": keepalive") is None
- assert backend._parse_sse_line("") is None
- assert backend._parse_sse_line("data: {bad json") is None
+ assert backend._parse_sse_line(": keepalive") == []
+ assert backend._parse_sse_line("") == []
+ assert backend._parse_sse_line("data: {bad json") == []
# An opening delta of {'role': 'assistant', 'content': None} carries
# nothing to render and must not be mistaken for end-of-stream.
opening = backend._parse_sse_line(
'data: {"choices":[{"delta":{"role":"assistant","content":null}}]}'
)
- assert opening is None
+ assert opening == [], opening
assert opening != backend.DONE
# A delta carrying both fields is thinking, not reply. Classifying it as
@@ -234,14 +234,30 @@ def test_sse_parsing():
'data: {"choices":[{"delta":'
'{"reasoning_content":"still thinking","content":""}}]}'
)
- assert both == ("reasoning", "still thinking"), both
+ assert both == [("reasoning", "still thinking")], both
# An empty reasoning delta renders nothing and must not fall through to
# the content branch.
empty = backend._parse_sse_line(
'data: {"choices":[{"delta":{"reasoning_content":""}}]}'
)
- assert empty is None, empty
+ assert empty == [], empty
+
+ # The final chunk of a tool round carries finish_reason and the usage
+ # block together; both must survive, or the tool call is dropped.
+ combined = backend._parse_sse_line(
+ 'data: {"choices":[{"index":0,"delta":{"content":"",'
+ '"reasoning_content":null},"finish_reason":"tool_calls"}],'
+ '"usage":{"prompt_tokens":388,"completion_tokens":74,'
+ '"total_tokens":462}}'
+ )
+ assert combined == [
+ (
+ "usage",
+ '{"prompt_tokens": 388, "completion_tokens": 74, "total_tokens": 462}',
+ ),
+ ("tool_finish", ""),
+ ], combined
print("ok sse parsing")
@@ -573,14 +589,16 @@ def test_usage_parsing():
'data: {"choices":[],"usage":{"prompt_tokens":1234,'
'"completion_tokens":56,"total_tokens":1290}}'
)
- kind, payload = backend._parse_sse_line(line)
+ events = backend._parse_sse_line(line)
+ assert [k for k, _ in events] == ["usage"], events
+ kind, payload = events[0]
assert kind == "usage"
stats = json.loads(payload)
assert stats["prompt_tokens"] == 1234
assert stats["total_tokens"] == 1290
# A usage-less chunk with empty choices is not mistaken for one.
- assert backend._parse_sse_line('data: {"choices":[]}') is None
+ assert backend._parse_sse_line('data: {"choices":[]}') == []
print("ok usage parsing")
@@ -928,6 +946,7 @@ def test_provider_parsing():
"price_in": 0.6,
"price_out": 0.9,
"thinking_budget": 8192,
+ "replay_reasoning": True,
},
}
}
@@ -941,6 +960,9 @@ def test_provider_parsing():
assert parsed["together"].price_in == 0.6
assert parsed["together"].price_out == 0.9
assert parsed["together"].thinking_budget == 8192
+ assert parsed["together"].replay_reasoning is True
+ # Off by default: replaying reasoning costs context and input tokens.
+ assert parsed["local"].replay_reasoning is False
# An old config: bare base_url, no providers table at all.
legacy = providers.parse({"base_url": "http://localhost:8181"})
@@ -1415,6 +1437,32 @@ def test_config_providers():
print("ok config provider table")
+def test_replays_reasoning():
+ """reasoning replay is per-provider, gated on search being on."""
+ from llamachat import providers
+
+ table = providers.parse(
+ {
+ "providers": {
+ "local": {"base_url": "http://localhost:8181"},
+ "deepseek": {"base_url": "https://api.deepseek.com",
+ "replay_reasoning": True},
+ "siliconflow": {"base_url": "https://api.siliconflow.com"},
+ }
+ }
+ )
+
+ # Only the opted-in provider replays, and only when tools are sent.
+ assert providers.replays_reasoning("deepseek:m", table, True) is True
+ assert providers.replays_reasoning("deepseek:m", table, False) is False
+ assert providers.replays_reasoning("siliconflow:m", table, True) is False
+ assert providers.replays_reasoning("localmodel", table, True) is False
+
+ # An unknown prefix resolves to local, which never replays.
+ assert providers.replays_reasoning("unconfigured:m", table, True) is False
+ print("ok replays reasoning")
+
+
def test_models_store():
"""models.ini round-trips per-model metadata and the cancel record."""
from llamachat import models
@@ -1829,6 +1877,123 @@ def test_token_column_migration():
print("ok token column migration")
+def test_usage_columns():
+ """Provider, raw usage and the GUI estimate persist per reply."""
+ with tempfile.TemporaryDirectory() as tmp:
+ history = db.History(Path(tmp) / "u.db")
+ sid = history.create_session("chat", "m", "t")
+ mid = history.add_message(sid, "assistant", "")
+ usage = '{"prompt_tokens":100,"completion_tokens":20,"total_tokens":120}'
+ history.update_message(
+ mid, "done",
+ prompt_tokens=100, completion_tokens=20,
+ model="siliconflow:zai-org/GLM-5.2", provider="siliconflow",
+ usage_json=usage, reported_cost_usd=0.000212,
+ )
+ row = history.messages(sid)[0]
+ assert row["provider"] == "siliconflow"
+ assert row["usage_json"] == usage
+ assert row["reported_cost_usd"] == 0.000212
+
+ # None means keep, exactly like the token columns.
+ history.update_message(mid, "edited")
+ kept = history.messages(sid)[0]
+ assert kept["provider"] == "siliconflow"
+ assert kept["usage_json"] == usage
+ assert kept["reported_cost_usd"] == 0.000212
+ history.close()
+ print("ok usage columns")
+
+
+def test_usage_accumulation():
+ """Every round of a searched turn reaches usage_all, not only the last.
+
+ Each search round is its own billed API call, so dropping the earlier
+ rounds' usage would under-report what a searched turn cost.
+ """
+ from llamachat import ui
+
+ class FakeClient:
+ def client_for(self, model):
+ return self
+
+ def wire_name(self, model):
+ return model
+
+ def stream_chat(self, wire, messages, search_cfg=None):
+ # One usage block per round: a tool round, then the final answer.
+ yield ("usage", '{"prompt_tokens": 100, "completion_tokens": 5,'
+ ' "total_tokens": 105}')
+ yield ("usage", '{"prompt_tokens": 140, "completion_tokens": 60,'
+ ' "total_tokens": 200}')
+
+ worker = ui.StreamWorker(FakeClient(), "m", [])
+ got: list[str] = []
+ worker.usage_all.connect(got.append)
+ worker.run()
+
+ assert len(got) == 1, got
+ blocks = json.loads(got[0])
+ assert [b["prompt_tokens"] for b in blocks] == [100, 140]
+ assert [b["completion_tokens"] for b in blocks] == [5, 60]
+
+ # A turn with no usage chunk at all (no include_usage support) yields
+ # an empty array, which the window stores as NULL.
+ class SilentClient(FakeClient):
+ def stream_chat(self, wire, messages, search_cfg=None):
+ yield ("content", "hi")
+ return
+
+ worker2 = ui.StreamWorker(SilentClient(), "m", [])
+ got2: list[str] = []
+ worker2.usage_all.connect(got2.append)
+ worker2.run()
+ assert got2 == ["[]"], got2
+ print("ok usage accumulation")
+
+
+def test_usage_column_migration():
+ """A pre-provider database opens, and the new columns read as NULL."""
+ import sqlite3
+
+ with tempfile.TemporaryDirectory() as tmp:
+ path = Path(tmp) / "old.db"
+ conn = sqlite3.connect(path)
+ conn.executescript(
+ "CREATE TABLE sessions (id INTEGER PRIMARY KEY, mode TEXT,"
+ " title TEXT, model TEXT, created_at INTEGER, updated_at INTEGER);"
+ "CREATE TABLE messages (id INTEGER PRIMARY KEY, session_id INTEGER,"
+ " role TEXT NOT NULL, content TEXT NOT NULL,"
+ " prompt_tokens INTEGER, completion_tokens INTEGER, model TEXT,"
+ " created_at INTEGER NOT NULL);"
+ "INSERT INTO sessions VALUES (1,'chat','old','m',0,0);"
+ "INSERT INTO messages VALUES (1,1,'assistant','older reply',"
+ "1200,340,'m',0);"
+ )
+ conn.commit()
+ conn.close()
+
+ history = db.History(path)
+ # The pre-migration row reads as unknown, never as an empty string
+ # or a fabricated zero.
+ old = history.messages(1)[0]
+ assert old["provider"] is None
+ assert old["usage_json"] is None
+ assert old["reported_cost_usd"] is None
+
+ mid = history.add_message(1, "assistant", "")
+ history.update_message(
+ mid, "new reply", prompt_tokens=10, completion_tokens=5,
+ model="q:r", provider="q", usage_json='{"total_tokens":15}',
+ )
+ fresh = history.messages(1)[1]
+ assert fresh["provider"] == "q"
+ assert fresh["usage_json"] == '{"total_tokens":15}'
+ assert fresh["reported_cost_usd"] is None
+ history.close()
+ print("ok usage column migration")
+
+
def test_client_auth_header():
"""A client with a key sends Bearer auth; one without sends no header."""
sent = {}
@@ -1908,6 +2073,33 @@ def test_client_auth_header():
print("ok client authorization header")
+def test_debug_log():
+ """The diagnostic log is off unless $LLAMACHAT_DEBUG_LOG names a path."""
+ with tempfile.TemporaryDirectory() as tmp:
+ path = Path(tmp) / "debug.log"
+ saved = os.environ.get("LLAMACHAT_DEBUG_LOG")
+ try:
+ # Unset -> a no-op that creates nothing to rotate.
+ os.environ.pop("LLAMACHAT_DEBUG_LOG", None)
+ backend._debug_log("must not be written")
+ assert not path.exists()
+
+ # Set -> appends timestamped records.
+ os.environ["LLAMACHAT_DEBUG_LOG"] = str(path)
+ backend._debug_log("first")
+ backend._debug_log("second")
+ lines = path.read_text(encoding="utf-8").strip().splitlines()
+ assert len(lines) == 2, lines
+ assert lines[0].startswith("[") and "first" in lines[0]
+ assert "second" in lines[1]
+ finally:
+ if saved is None:
+ os.environ.pop("LLAMACHAT_DEBUG_LOG", None)
+ else:
+ os.environ["LLAMACHAT_DEBUG_LOG"] = saved
+ print("ok debug log")
+
+
def test_stream_body_for_cloud():
"""Cloud providers get max_tokens and thinking_budget; local does not."""
captured = {}
@@ -2507,6 +2699,115 @@ def test_search_failure_paths():
print("ok search failure paths")
+def test_reasoning_content_preserved():
+ """A tool-call round replays its reasoning_content verbatim.
+
+ DeepSeek V3.2+/V4 and GLM-4.7+ on SiliconFlow emit chain-of-thought as
+ reasoning_content and require it sent back unchanged on the assistant
+ tool-call message. Dropping it breaks their multi-step tool flow, which
+ surfaces as a turn that stops at the thinking with no answer.
+ """
+ import urllib.request
+
+ from llamachat import search
+
+ client = backend.Client("http://x")
+ seen_messages = []
+
+ def scripted(model, messages, tools):
+ seen_messages.append([dict(m) for m in messages])
+ if len(seen_messages) == 1:
+ yield ("reasoning", "I should check the current version.")
+ yield from _tool_round("latest kernel")
+ else:
+ yield ("content", "answered")
+
+ client._stream_once = scripted
+ original = _with_urlopen(json.dumps({"results": []}).encode())
+ try:
+ cfg = backend.SearchConfig(
+ enabled=True, url="http://searx", max_searches=1
+ )
+ list(client.stream_chat("m", [{"role": "user", "content": "hi"}], cfg))
+ finally:
+ urllib.request.urlopen = original
+
+ # The second request's assistant message must carry the thinking back.
+ second = seen_messages[1]
+ assistant = next(m for m in second if m["role"] == "assistant" and m.get("tool_calls"))
+ assert assistant["reasoning_content"] == "I should check the current version.", assistant
+
+ # The reasoning still reaches the UI unchanged.
+ assert len(seen_messages) == 2, seen_messages
+ print("ok reasoning content preserved")
+
+
+def test_tool_call_finish_with_usage():
+ """finish_reason riding the usage chunk still triggers the search.
+
+ DeepSeek's include_usage stream puts finish_reason: "tool_calls" on the
+ same SSE line as the usage block. The old single-event parser returned
+ that line as "usage" and never saw the finish, so the tool call was
+ dropped and the turn ended at the thinking with no search.
+ """
+ import httpx
+ import urllib.request
+
+ def sse(obj):
+ return "data: " + json.dumps(obj)
+
+ lines = [
+ sse({"choices": [{"delta": {"reasoning_content": "hmm"}}]}),
+ sse({"choices": [{"delta": {"tool_calls": [
+ {
+ "index": 0,
+ "id": "call_1",
+ "type": "function",
+ "function": {"name": "web_search", "arguments": '{"query":"kernel"}'},
+ }
+ ]}}]}),
+ sse({
+ "choices": [{
+ "index": 0,
+ "delta": {"content": "", "reasoning_content": None},
+ "finish_reason": "tool_calls",
+ }],
+ "usage": {"prompt_tokens": 10, "completion_tokens": 5, "total_tokens": 15},
+ }),
+ "data: [DONE]",
+ ]
+
+ class _Stream:
+ status_code = 200
+
+ def __enter__(self):
+ return self
+
+ def __exit__(self, *exc):
+ return False
+
+ def iter_lines(self):
+ return iter(lines)
+
+ def _fake_stream(method, url, json=None, timeout=None, headers=None):
+ return _Stream()
+
+ client = backend.Client("http://x")
+ original = _with_urlopen(json.dumps({"results": []}).encode())
+ try:
+ with _patched(httpx, "stream", _fake_stream):
+ cfg = backend.SearchConfig(enabled=True, url="http://searx", max_searches=1)
+ out = list(client.stream_chat("m", [{"role": "user", "content": "hi"}], cfg))
+ finally:
+ urllib.request.urlopen = original
+
+ kinds = [k for k, _ in out]
+ assert "search_start" in kinds, kinds
+ assert "search_done" in kinds, kinds
+ assert ("reasoning", "hmm") in out, out
+ print("ok tool call finish with usage")
+
+
def test_search_storage():
import sqlite3
@@ -2877,10 +3178,15 @@ if __name__ == "__main__":
test_model_ids_and_filtering()
test_key_resolution()
test_config_providers()
+ test_replays_reasoning()
test_models_store()
test_metadata_and_cost()
test_token_column_migration()
+ test_usage_columns()
+ test_usage_accumulation()
+ test_usage_column_migration()
test_client_auth_header()
+ test_debug_log()
test_stream_body_for_cloud()
test_multi_client()
test_model_dialog_values()
@@ -2890,6 +3196,8 @@ if __name__ == "__main__":
test_tool_call_accumulation()
test_search_loop_cap()
test_search_failure_paths()
+ test_reasoning_content_preserved()
+ test_tool_call_finish_with_usage()
test_search_storage()
test_date_note()
test_search_html()