diff options
| author | Danilo M. <danix@danix.xyz> | 2026-08-21 20:58:39 +0200 |
|---|---|---|
| committer | Danilo M. <danix@danix.xyz> | 2026-08-21 20:58:39 +0200 |
| commit | a7422ddeb7771983e984350b31092fe4898897c6 (patch) | |
| tree | 2833a9c8f5c2ab646ceee5f42b711b7d0c35ec47 | |
| parent | 2cae7f19ac90e6df9d4008e6e364bae30e8b389a (diff) | |
| download | llamachat-feature/external-providers.tar.gz llamachat-feature/external-providers.zip | |
fix: web search on DeepSeek and other reasoning modelsfeature/external-providers
A searched turn on a cloud reasoning model ended at the thinking: the
model emitted the tool call, but finish_reason: "tool_calls" landed on
the same SSE line as the include_usage block, so the single-event parser
returned that line as a usage chunk and the loop never saw the tool
finish. The parser now emits every event a line carries, so the search
fires.
Also in this change:
- Replay each round's reasoning_content on the assistant tool-call
message, which interleaved-thinking models require to keep going.
- Add a per-provider replay_reasoning option (DeepSeek, SiliconFlow
GLM-4.7+) to carry prior turns' reasoning_content when search is on.
- Add an on-demand diagnostic log gated by $LLAMACHAT_DEBUG_LOG.
- Record provider, usage_json and reported_cost_usd per reply, so a
searched turn keeps every round's billed usage for external consumers.
| -rw-r--r-- | CHANGELOG.md | 27 | ||||
| -rw-r--r-- | README.md | 28 | ||||
| -rw-r--r-- | llamachat/backend.py | 118 | ||||
| -rw-r--r-- | llamachat/config.py | 8 | ||||
| -rw-r--r-- | llamachat/db.py | 24 | ||||
| -rw-r--r-- | llamachat/providers.py | 22 | ||||
| -rw-r--r-- | llamachat/ui.py | 76 | ||||
| -rwxr-xr-x | test_llamachat.py | 328 |
8 files changed, 584 insertions, 47 deletions
diff --git a/CHANGELOG.md b/CHANGELOG.md index 6e9a831..34314f8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -34,10 +34,28 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 `model`. Token counts come from the stream's final usage chunk and power the cost readout; the model column records which model actually replied, so a switched conversation prices each turn correctly. +- Three more columns on `messages`: `provider`, `usage_json` and + `reported_cost_usd`. The provider name is recorded when a reply finishes so + history stays self-describing even if a provider is later renamed or + removed; `usage_json` keeps every round's raw usage block from the stream + as a JSON array (a searched turn makes one API call per search round, and + all of them are billed, so all of them survive) for external consumers + such as the cost dashboard's ingester; and `reported_cost_usd` stores the + GUI's hand-entered-price estimate for the reply, summed across rounds, as + a sanity check only, staying NULL for an unpriced model rather than + claiming it cost nothing. - Per-provider `thinking_budget` option, for endpoints (currently SiliconFlow) that cap chain-of-thought tokens separately from the final answer. Unset providers skip the key entirely so other endpoints do not receive an unknown parameter. +- Per-provider `replay_reasoning` option. When set and web search is enabled, + each prior assistant turn's `reasoning_content` is replayed on the next + request, which interleaved-thinking providers (DeepSeek, SiliconFlow's + GLM-4.7+) require when a `tools` key is present. Off by default so other + providers do not pay context and input tokens for thinking they ignore. +- On-demand diagnostic log. Setting `LLAMACHAT_DEBUG_LOG` to a path appends + timestamped provider-error records (status, request body, full response) to + that file; unset, it writes nothing, so a normal launch never grows a log. ### Fixed @@ -48,6 +66,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 which could be too small for a reasoning model. Non-local providers now send `max_tokens = 32768`, leaving room for both chain-of-thought reasoning and the answer. +- A streamed tool call is now detected even when the provider puts + `finish_reason: "tool_calls"` on the same SSE line as the usage block (the + `include_usage` final chunk). The parser previously returned that line as + `usage` and dropped the finish signal, so a DeepSeek web search never + triggered and the turn ended at the thinking. +- A searched turn now replays the model's `reasoning_content` verbatim on the + assistant tool-call message. Interleaved-thinking models (DeepSeek V3.2+/V4, + GLM-4.7+ on SiliconFlow) require this and stop answering — ending the turn at + the thinking — when it is dropped. Tool calling and reasoning output vary between providers. Web search on a cloud model may not work as reliably as it does with a local router. @@ -288,7 +288,9 @@ there is no second thread. ```sql sessions (id, mode, title, model, prompt_name, prompt_custom, created_at, updated_at) -messages (id, session_id, role, content, reasoning, searches, created_at) +messages (id, session_id, role, content, reasoning, searches, + prompt_tokens, completion_tokens, model, provider, usage_json, + reported_cost_usd, created_at) attachments (id, message_id, path, kind, mime, size, sha256, thumb, truncated) messages_fts -- FTS5 external-content table over messages.content ``` @@ -307,6 +309,19 @@ per search with its query, results and any error, or NULL when nothing was searched. Like `reasoning` it stays out of the FTS index, or snippets from web pages would compete with messages the user actually wrote. +`provider` records which provider served the reply (`local` or a configured +provider name), written when the reply finishes so history stays +self-describing even if `config.toml` later renames or drops the provider. +`usage_json` keeps every round's raw usage block from the stream as a JSON +array — a searched turn makes one API call per search round, and each round +is billed separately, so all of them are kept verbatim and token details +beyond the two counts (e.g. cached or reasoning tokens on a cloud endpoint) +are recoverable later without re-guessing a field mapping. +`reported_cost_usd` is the GUI's hand-entered-price estimate for that one +reply (summed across rounds), stored as a sanity check for external +consumers — it is approximate and never a source of truth; it stays NULL for +a model with no entered prices rather than claiming the reply cost nothing. + `prompt_name` records which system prompt a conversation was built with, and `prompt_custom` holds the text when that prompt is a one-off rather than a file. Storing the name rather than the resolved text means editing a preset @@ -463,6 +478,8 @@ filter = ["qwen", "deepseek"] ctx_size = 32768 price_in = 0.60 price_out = 0.60 +# thinking_budget = 8192 # per-provider chain-of-thought cap +# replay_reasoning = false # see below ``` `api_key` accepts three forms: @@ -499,6 +516,15 @@ local router, but on a cloud provider every message is billed for the entire conversation so far. The projection in the cost label exists to make that visible before you send. +`thinking_budget` sets a provider-specific cap on chain-of-thought tokens +(currently sent to SiliconFlow; other endpoints ignore it when unset). + +`replay_reasoning` makes each assistant turn's thinking be sent back on the +next request while web search is enabled. DeepSeek's and SiliconFlow's +interleaved-thinking models require this when a `tools` key is present, and +return a 400 if it is missing. Leave it off elsewhere — replaying thinking +costs context and input tokens for providers that ignore it. + Tool calling and reasoning output vary between providers. Web search on a cloud model may not work as reliably as it does with a local router. diff --git a/llamachat/backend.py b/llamachat/backend.py index 4f15936..338f396 100644 --- a/llamachat/backend.py +++ b/llamachat/backend.py @@ -18,7 +18,9 @@ import hashlib import io import json import mimetypes +import os from dataclasses import dataclass, field +from datetime import datetime from pathlib import Path import httpx @@ -46,6 +48,26 @@ IMAGE_MIMES = {"image/png", "image/jpeg", "image/gif", "image/webp", "image/bmp" DONE = ("done", "") +def _debug_log(entry: str) -> None: + """Append one diagnostic record to $LLAMACHAT_DEBUG_LOG, when set. + + On-demand by design: unset (the default) this is a no-op and no file is + ever created, so a normal launch never grows a log to rotate. Set the + variable to a path to capture provider errors while chasing one, then + unset it again. The request body is safe to log: the API key travels in + the Authorization header, never in the JSON body. + """ + path = os.environ.get("LLAMACHAT_DEBUG_LOG") + if not path: + return + stamp = datetime.now().isoformat(timespec="seconds") + try: + with open(path, "a", encoding="utf-8") as fh: + fh.write(f"[{stamp}] {entry}\n") + except OSError: + pass # a bad log path must never take down the chat + + class BackendError(Exception): """Any failure talking to the router, already phrased for the user.""" @@ -206,8 +228,10 @@ class Client: resp.raise_for_status() payload = resp.json() except httpx.HTTPError as exc: + _debug_log(f"GET /v1/models failed: {exc}") raise BackendError(f"Cannot reach router at {self.base_url}: {exc}") except ValueError as exc: + _debug_log(f"GET /v1/models sent invalid JSON: {exc}") raise BackendError(f"Router sent invalid JSON: {exc}") # Some OpenAI-compatible providers return a bare array instead of # the standard {"data": [...]} envelope. Accept both. @@ -241,8 +265,10 @@ class Client: resp.raise_for_status() payload = resp.json() except httpx.HTTPError as exc: + _debug_log(f"POST /v1/chat/completions (complete) failed: {exc}") raise BackendError(f"Request failed: {exc}") except ValueError as exc: + _debug_log(f"POST /v1/chat/completions (complete) invalid JSON: {exc}") raise BackendError(f"Router sent invalid JSON: {exc}") choices = payload.get("choices") or [] if not choices: @@ -280,6 +306,7 @@ class Client: tools = None if last_round else [search.TOOL_SCHEMA] calls: dict = {} saw_tool_finish = False + reasoning = "" for kind, piece in self._stream_once(model, convo, tools=tools): if kind == "tool_calls": @@ -287,6 +314,8 @@ class Client: elif kind == "tool_finish": saw_tool_finish = True else: + if kind == "reasoning": + reasoning += piece yield (kind, piece) wanted = [ @@ -297,23 +326,30 @@ class Client: if not (saw_tool_finish and wanted) or last_round: return - convo.append( - { - "role": "assistant", - "content": None, - "tool_calls": [ - { - "id": call["id"], - "type": "function", - "function": { - "name": call["name"], - "arguments": call["arguments"], - }, - } - for call in wanted - ], - } - ) + assistant = { + "role": "assistant", + "content": None, + "tool_calls": [ + { + "id": call["id"], + "type": "function", + "function": { + "name": call["name"], + "arguments": call["arguments"], + }, + } + for call in wanted + ], + } + # Interleaved-thinking models (DeepSeek V3.2+/V4, GLM-4.7+ on + # SiliconFlow) emit their chain-of-thought as reasoning_content and + # require it replayed verbatim on the assistant tool-call message. + # Dropping it breaks their multi-step tool flow: the follow-up + # round ends with the model thinking and no answer. Local reasoning + # models tolerate the extra key, so include it whenever it exists. + if reasoning: + assistant["reasoning_content"] = reasoning + convo.append(assistant) # The round after this one is the last, so its results are the # last the model will get. Saying so is what makes it answer # instead of asking for a search it can no longer make. @@ -392,17 +428,21 @@ class Client: ) as resp: if resp.status_code != 200: resp.read() + _debug_log( + f"POST /v1/chat/completions -> {resp.status_code}\n" + f"request: {json.dumps(body, ensure_ascii=False)}\n" + f"response: {resp.text}" + ) raise BackendError( f"Router returned {resp.status_code}: {resp.text[:300]}" ) for line in resp.iter_lines(): chunk = _parse_sse_line(line) - if chunk is None: - continue if chunk == DONE: return - yield chunk + yield from chunk except httpx.HTTPError as exc: + _debug_log(f"POST /v1/chat/completions transport error: {exc}") raise BackendError(f"Request failed: {exc}") @@ -588,46 +628,54 @@ def estimate_tokens(messages: list[dict], chars_per_token: float) -> int: return int(chars / max(chars_per_token, 1.0)) + overhead + images * 600 -def _parse_sse_line(line: str) -> tuple[str, str] | None: - """Decode one SSE line. +def _parse_sse_line(line: str): + """Decode one SSE line into zero or more (kind, text) events. - Returns a ('reasoning'|'content', text) pair, the DONE sentinel at the - end of the stream, or None for lines carrying nothing to render. + Returns a list, empty for lines carrying nothing to render, or the DONE + sentinel at the end of the stream. One line can carry two events: with + `stream_options.include_usage` the final chunk holds both the + finish_reason and the usage block together, and a searched turn needs + both — the former marks the tool call, the latter its token counts. """ if not line.startswith("data:"): - return None + return [] payload = line[5:].strip() if payload == "[DONE]": return DONE try: obj = json.loads(payload) except ValueError: - return None + return [] + + events: list[tuple[str, str]] = [] + # The usage chunk arrives with an empty choices list, so check it first. usage = obj.get("usage") if usage and usage.get("prompt_tokens") is not None: - return ("usage", json.dumps(usage)) + events.append(("usage", json.dumps(usage))) choices = obj.get("choices") or [] if not choices: - return None + return events delta = choices[0].get("delta") or {} # Tool-call fragments arrive in their own delta field, alongside a # finish_reason of 'tool_calls' on the final chunk of the round. tool_calls = delta.get("tool_calls") if tool_calls: - return ("tool_calls", json.dumps(tool_calls)) + events.append(("tool_calls", json.dumps(tool_calls))) if choices[0].get("finish_reason") == "tool_calls": - return ("tool_finish", "") + events.append(("tool_finish", "")) # Both fields can arrive in one delta, and a reasoning delta can be an # empty string. Test for presence, not truthiness, so a chunk carrying # reasoning is never mistaken for a content chunk. reasoning = delta.get("reasoning_content") if reasoning is not None: - return ("reasoning", reasoning) if reasoning else None - content = delta.get("content") - if content: - return ("content", content) - return None + if reasoning: + events.append(("reasoning", reasoning)) + else: + content = delta.get("content") + if content: + events.append(("content", content)) + return events diff --git a/llamachat/config.py b/llamachat/config.py index 81b799b..0c119ad 100644 --- a/llamachat/config.py +++ b/llamachat/config.py @@ -253,6 +253,13 @@ def write_default(path: Path = CONFIG_PATH) -> Path: '# default is 4096; raise it for complex questions that need more\n' '# room to reason.\n' '#\n' + '# replay_reasoning makes each assistant turn\'s thinking be sent back\n' + '# on the next request while web search is enabled. DeepSeek and\n' + '# SiliconFlow\'s interleaved-thinking models reject a tools request\n' + '# whose prior assistant turns omit reasoning_content, so set it for\n' + '# those; leave it off elsewhere, since replaying thinking costs\n' + '# context and input tokens.\n' + '#\n' '# [providers.together]\n' '# base_url = "https://api.together.xyz"\n' '# api_key = "pass:api/together"\n' @@ -261,6 +268,7 @@ def write_default(path: Path = CONFIG_PATH) -> Path: '# price_in = 0.60\n' '# price_out = 0.60\n' '# thinking_budget = 8192\n' + '# replay_reasoning = false\n' ) return path diff --git a/llamachat/db.py b/llamachat/db.py index e88c131..9ec220a 100644 --- a/llamachat/db.py +++ b/llamachat/db.py @@ -42,6 +42,9 @@ CREATE TABLE IF NOT EXISTS messages ( prompt_tokens INTEGER, completion_tokens INTEGER, model TEXT, + provider TEXT, -- provider name at write time ('local' or a configured one) + usage_json TEXT, -- raw usage block from the stream's final SSE chunk + reported_cost_usd REAL, -- GUI's hand-entered-price estimate; sanity check only created_at INTEGER NOT NULL ); @@ -117,6 +120,21 @@ class History: # The model on `sessions` is the current one, which prices a # switched conversation wrongly. Record what actually replied. "model": "TEXT", + # Nullable: pre-migration rows have no provider recorded, and + # the ingester's ':'-split fallback covers those. Storing it + # at write time beats re-deriving it from config.toml later, + # which the user may rename or remove. + "provider": "TEXT", + # The raw usage block from the stream's final SSE chunk, kept + # verbatim like the dashboard's raw_json: a future field + # mapping (e.g. cached tokens) is re-derivable, not guessed. + "usage_json": "TEXT", + # The GUI's hand-entered-price estimate for this reply. A + # sanity check only -- real numbers come from the dashboard's + # pricing engine. NULL (unpriced model) is never written as + # 0.0: zero would claim a reply cost nothing when the GUI + # simply could not say. + "reported_cost_usd": "REAL", }, "sessions": { "prompt_name": "TEXT NOT NULL DEFAULT ''", @@ -226,6 +244,9 @@ class History: prompt_tokens: int | None = None, completion_tokens: int | None = None, model: str | None = None, + provider: str | None = None, + usage_json: str | None = None, + reported_cost_usd: float | None = None, ) -> None: """Fill in a streamed reply. Omitted fields keep their stored value. @@ -242,6 +263,9 @@ class History: ("prompt_tokens", prompt_tokens), ("completion_tokens", completion_tokens), ("model", model), + ("provider", provider), + ("usage_json", usage_json), + ("reported_cost_usd", reported_cost_usd), ): if value is not None: columns.append(f"{column} = ?") diff --git a/llamachat/providers.py b/llamachat/providers.py index a8b2350..23ad081 100644 --- a/llamachat/providers.py +++ b/llamachat/providers.py @@ -49,6 +49,12 @@ class Provider: # Provider-specific reasoning token budget. Currently SiliconFlow only; # ignored when unset so other endpoints do not receive an unknown key. thinking_budget: int | None = None + # Replay each assistant turn's reasoning_content on the next request. + # Interleaved-thinking providers (DeepSeek, SiliconFlow's GLM-4.7+) reject + # a tools request whose prior assistant turns omit it. Off by default: + # replaying thinking inflates context and the input bill for providers + # that neither need nor want it. + replay_reasoning: bool = False @property def is_local(self) -> bool: @@ -189,6 +195,7 @@ def parse(values: dict, warnings: list[str] | None = None) -> dict[str, Provider price_in=_number(entry.get("price_in"), float), price_out=_number(entry.get("price_out"), float), thinking_budget=_number(entry.get("thinking_budget"), int), + replay_reasoning=bool(entry.get("replay_reasoning", False)), ) return out @@ -220,6 +227,21 @@ def split(model_id: str, table: dict[str, Provider]) -> tuple[str, str]: return LOCAL, model_id +def replays_reasoning(model_id: str, table: dict[str, Provider], search_enabled: bool) -> bool: + """Whether a request to this model must replay reasoning_content. + + Only true when search is enabled, because that is the only time the app + sends a `tools` key — the condition under which interleaved-thinking + providers demand the reasoning back. The caller still checks that a given + row has non-empty reasoning; a non-reasoning model has nothing to replay. + """ + if not search_enabled: + return False + name, _ = split(model_id, table) + provider = table.get(name) + return bool(provider and provider.replay_reasoning) + + def apply_filter(provider: Provider, listed: list[str]) -> list[str]: """Keep models matching any of the provider's substrings. diff --git a/llamachat/ui.py b/llamachat/ui.py index 189d59a..32f7242 100644 --- a/llamachat/ui.py +++ b/llamachat/ui.py @@ -193,6 +193,7 @@ class StreamWorker(QObject): chunk = Signal(str) reasoning = Signal(str) usage = Signal(int, int) + usage_all = Signal(str) # every round's usage block, as a JSON array search_start = Signal(str) search_done = Signal(str) finished = Signal() @@ -211,6 +212,7 @@ class StreamWorker(QObject): self.messages = messages self.search_cfg = search_cfg self._stop = False + self._usage_blocks: list[str] = [] def stop(self) -> None: self._stop = True @@ -243,6 +245,11 @@ class StreamWorker(QObject): self.search_done.emit(piece) elif kind == "usage": stats = json.loads(piece) + # A searched turn makes several API calls (one per search + # round); every round's usage block must survive for the + # dashboard to price each billed request, not just the + # last one the meter shows. + self._usage_blocks.append(piece) self.usage.emit( int(stats.get("prompt_tokens") or 0), int(stats.get("total_tokens") or 0), @@ -255,6 +262,9 @@ class StreamWorker(QObject): except Exception as exc: # noqa: BLE001 - surface anything as UI text self.failed.emit(f"Unexpected error: {exc}") return + self.usage_all.emit( + json.dumps([json.loads(b) for b in self._usage_blocks]) + ) self.finished.emit() @@ -461,6 +471,7 @@ class ChatWindow(QMainWindow): self.searches: list[dict] = [] self.turn_prompt_tokens: int | None = None self.turn_completion_tokens: int | None = None + self.turn_usage_json: str | None = None self._conversation_cost = 0.0 # Every rendered bubble, so a reasoning toggle can redraw the # transcript without refetching anything. @@ -1127,8 +1138,18 @@ class ChatWindow(QMainWindow): """Prior turns plus the new message, for multi-turn mode.""" messages = self._system_messages() rows = self.history.messages(self.session_id) + # Interleaved-thinking providers require each assistant turn's + # reasoning_content back on the next tools request (see + # providers.replays_reasoning). Non-empty only, so a plain model + # never carries an empty key it never produced. + replay = providers_mod.replays_reasoning( + self.current_model(), self.cfg.providers, self.cfg.search_enabled + ) for row in rows[:-1]: # the latest user row is replaced below - messages.append({"role": row["role"], "content": row["content"]}) + message = {"role": row["role"], "content": row["content"]} + if replay and row["role"] == "assistant" and row["reasoning"]: + message["reasoning_content"] = row["reasoning"] + messages.append(message) messages.append({"role": "user", "content": latest_content}) return messages @@ -1240,6 +1261,7 @@ class ChatWindow(QMainWindow): self.worker.search_start.connect(self._on_search_start) self.worker.search_done.connect(self._on_search_done) self.worker.usage.connect(self._on_usage) + self.worker.usage_all.connect(self._on_usage_all) self.worker.finished.connect(self._on_stream_finished) self.worker.failed.connect(self._on_stream_failed) self.thread.start() @@ -1264,6 +1286,18 @@ class ChatWindow(QMainWindow): self.update_cost() @Slot(str) + def _on_usage_all(self, blocks: str) -> None: + """Keep every round's usage block for the finished turn. + + A searched turn makes several API calls (one per search round) and + the dashboard prices each round's tokens, so all of them must + survive -- not just the last one the meter shows. The block is a + JSON array of the raw per-round usage objects; the ingester sums + what it needs. + """ + self.turn_usage_json = blocks if blocks != "[]" else None + + @Slot(str) def _on_reasoning(self, piece: str) -> None: if not self.reasoning_buffer: self.hide_status() @@ -1289,6 +1323,42 @@ class ChatWindow(QMainWindow): """Stored form of this turn's searches: NULL when there were none.""" return json.dumps(self.searches) if self.searches else None + def _current_provider(self) -> str: + """The provider that served the current model, as stored. + + 'local' for a bare model id, the configured provider name otherwise + (providers.split resolves against the provider table). Stored per + reply so history stays self-describing even if config.toml later + renames or drops the provider. + """ + return providers_mod.split(self.current_model(), self.cfg.providers)[0] + + def _turn_reported_cost(self) -> float | None: + """The GUI's hand-entered-price estimate for the finished reply. + + NULL for a model with no entered prices (is_priced): 0.0 would + claim the reply cost nothing when the GUI simply cannot say. The + dashboard treats this as a sanity check only, never as a source of + truth. Sums every round of a searched turn: each round is its own + billed request. + """ + model_id = self.current_model() + if not models_mod.is_priced(model_id, self.cfg.providers, self.store): + return None + prompt = 0 + completion = 0 + for block in json.loads(self.turn_usage_json or "[]"): + prompt += block.get("prompt_tokens") or 0 + completion += block.get("completion_tokens") or 0 + if not (prompt or completion): + return None + row = { + "model": model_id, + "prompt_tokens": prompt, + "completion_tokens": completion, + } + return models_mod.message_cost(row, self.cfg.providers, self.store) + @Slot() def _on_stream_finished(self) -> None: if self.assistant_message_id is not None: @@ -1300,6 +1370,9 @@ class ChatWindow(QMainWindow): prompt_tokens=self.turn_prompt_tokens, completion_tokens=self.turn_completion_tokens, model=self.current_model(), + provider=self._current_provider(), + usage_json=self.turn_usage_json, + reported_cost_usd=self._turn_reported_cost(), ) rows = self.history.messages(self.session_id) self._conversation_cost = models_mod.conversation_cost( @@ -1382,6 +1455,7 @@ class ChatWindow(QMainWindow): self.assistant_message_id = None self.turn_prompt_tokens = None self.turn_completion_tokens = None + self.turn_usage_json = None self.send_button.setEnabled(True) self.stop_button.hide() diff --git a/test_llamachat.py b/test_llamachat.py index 3f1f678..74b0d0c 100755 --- a/test_llamachat.py +++ b/test_llamachat.py @@ -208,24 +208,24 @@ def test_classify(): def test_sse_parsing(): line = 'data: {"choices":[{"delta":{"content":"hi"}}]}' - assert backend._parse_sse_line(line) == ("content", "hi") + assert backend._parse_sse_line(line) == [("content", "hi")] # Reasoning arrives in its own field, which is what lets the UI keep # thinking and reply apart without parsing <think> tags. think = 'data: {"choices":[{"delta":{"reasoning_content":"hmm"}}]}' - assert backend._parse_sse_line(think) == ("reasoning", "hmm") + assert backend._parse_sse_line(think) == [("reasoning", "hmm")] assert backend._parse_sse_line("data: [DONE]") == backend.DONE - assert backend._parse_sse_line(": keepalive") is None - assert backend._parse_sse_line("") is None - assert backend._parse_sse_line("data: {bad json") is None + assert backend._parse_sse_line(": keepalive") == [] + assert backend._parse_sse_line("") == [] + assert backend._parse_sse_line("data: {bad json") == [] # An opening delta of {'role': 'assistant', 'content': None} carries # nothing to render and must not be mistaken for end-of-stream. opening = backend._parse_sse_line( 'data: {"choices":[{"delta":{"role":"assistant","content":null}}]}' ) - assert opening is None + assert opening == [], opening assert opening != backend.DONE # A delta carrying both fields is thinking, not reply. Classifying it as @@ -234,14 +234,30 @@ def test_sse_parsing(): 'data: {"choices":[{"delta":' '{"reasoning_content":"still thinking","content":""}}]}' ) - assert both == ("reasoning", "still thinking"), both + assert both == [("reasoning", "still thinking")], both # An empty reasoning delta renders nothing and must not fall through to # the content branch. empty = backend._parse_sse_line( 'data: {"choices":[{"delta":{"reasoning_content":""}}]}' ) - assert empty is None, empty + assert empty == [], empty + + # The final chunk of a tool round carries finish_reason and the usage + # block together; both must survive, or the tool call is dropped. + combined = backend._parse_sse_line( + 'data: {"choices":[{"index":0,"delta":{"content":"",' + '"reasoning_content":null},"finish_reason":"tool_calls"}],' + '"usage":{"prompt_tokens":388,"completion_tokens":74,' + '"total_tokens":462}}' + ) + assert combined == [ + ( + "usage", + '{"prompt_tokens": 388, "completion_tokens": 74, "total_tokens": 462}', + ), + ("tool_finish", ""), + ], combined print("ok sse parsing") @@ -573,14 +589,16 @@ def test_usage_parsing(): 'data: {"choices":[],"usage":{"prompt_tokens":1234,' '"completion_tokens":56,"total_tokens":1290}}' ) - kind, payload = backend._parse_sse_line(line) + events = backend._parse_sse_line(line) + assert [k for k, _ in events] == ["usage"], events + kind, payload = events[0] assert kind == "usage" stats = json.loads(payload) assert stats["prompt_tokens"] == 1234 assert stats["total_tokens"] == 1290 # A usage-less chunk with empty choices is not mistaken for one. - assert backend._parse_sse_line('data: {"choices":[]}') is None + assert backend._parse_sse_line('data: {"choices":[]}') == [] print("ok usage parsing") @@ -928,6 +946,7 @@ def test_provider_parsing(): "price_in": 0.6, "price_out": 0.9, "thinking_budget": 8192, + "replay_reasoning": True, }, } } @@ -941,6 +960,9 @@ def test_provider_parsing(): assert parsed["together"].price_in == 0.6 assert parsed["together"].price_out == 0.9 assert parsed["together"].thinking_budget == 8192 + assert parsed["together"].replay_reasoning is True + # Off by default: replaying reasoning costs context and input tokens. + assert parsed["local"].replay_reasoning is False # An old config: bare base_url, no providers table at all. legacy = providers.parse({"base_url": "http://localhost:8181"}) @@ -1415,6 +1437,32 @@ def test_config_providers(): print("ok config provider table") +def test_replays_reasoning(): + """reasoning replay is per-provider, gated on search being on.""" + from llamachat import providers + + table = providers.parse( + { + "providers": { + "local": {"base_url": "http://localhost:8181"}, + "deepseek": {"base_url": "https://api.deepseek.com", + "replay_reasoning": True}, + "siliconflow": {"base_url": "https://api.siliconflow.com"}, + } + } + ) + + # Only the opted-in provider replays, and only when tools are sent. + assert providers.replays_reasoning("deepseek:m", table, True) is True + assert providers.replays_reasoning("deepseek:m", table, False) is False + assert providers.replays_reasoning("siliconflow:m", table, True) is False + assert providers.replays_reasoning("localmodel", table, True) is False + + # An unknown prefix resolves to local, which never replays. + assert providers.replays_reasoning("unconfigured:m", table, True) is False + print("ok replays reasoning") + + def test_models_store(): """models.ini round-trips per-model metadata and the cancel record.""" from llamachat import models @@ -1829,6 +1877,123 @@ def test_token_column_migration(): print("ok token column migration") +def test_usage_columns(): + """Provider, raw usage and the GUI estimate persist per reply.""" + with tempfile.TemporaryDirectory() as tmp: + history = db.History(Path(tmp) / "u.db") + sid = history.create_session("chat", "m", "t") + mid = history.add_message(sid, "assistant", "") + usage = '{"prompt_tokens":100,"completion_tokens":20,"total_tokens":120}' + history.update_message( + mid, "done", + prompt_tokens=100, completion_tokens=20, + model="siliconflow:zai-org/GLM-5.2", provider="siliconflow", + usage_json=usage, reported_cost_usd=0.000212, + ) + row = history.messages(sid)[0] + assert row["provider"] == "siliconflow" + assert row["usage_json"] == usage + assert row["reported_cost_usd"] == 0.000212 + + # None means keep, exactly like the token columns. + history.update_message(mid, "edited") + kept = history.messages(sid)[0] + assert kept["provider"] == "siliconflow" + assert kept["usage_json"] == usage + assert kept["reported_cost_usd"] == 0.000212 + history.close() + print("ok usage columns") + + +def test_usage_accumulation(): + """Every round of a searched turn reaches usage_all, not only the last. + + Each search round is its own billed API call, so dropping the earlier + rounds' usage would under-report what a searched turn cost. + """ + from llamachat import ui + + class FakeClient: + def client_for(self, model): + return self + + def wire_name(self, model): + return model + + def stream_chat(self, wire, messages, search_cfg=None): + # One usage block per round: a tool round, then the final answer. + yield ("usage", '{"prompt_tokens": 100, "completion_tokens": 5,' + ' "total_tokens": 105}') + yield ("usage", '{"prompt_tokens": 140, "completion_tokens": 60,' + ' "total_tokens": 200}') + + worker = ui.StreamWorker(FakeClient(), "m", []) + got: list[str] = [] + worker.usage_all.connect(got.append) + worker.run() + + assert len(got) == 1, got + blocks = json.loads(got[0]) + assert [b["prompt_tokens"] for b in blocks] == [100, 140] + assert [b["completion_tokens"] for b in blocks] == [5, 60] + + # A turn with no usage chunk at all (no include_usage support) yields + # an empty array, which the window stores as NULL. + class SilentClient(FakeClient): + def stream_chat(self, wire, messages, search_cfg=None): + yield ("content", "hi") + return + + worker2 = ui.StreamWorker(SilentClient(), "m", []) + got2: list[str] = [] + worker2.usage_all.connect(got2.append) + worker2.run() + assert got2 == ["[]"], got2 + print("ok usage accumulation") + + +def test_usage_column_migration(): + """A pre-provider database opens, and the new columns read as NULL.""" + import sqlite3 + + with tempfile.TemporaryDirectory() as tmp: + path = Path(tmp) / "old.db" + conn = sqlite3.connect(path) + conn.executescript( + "CREATE TABLE sessions (id INTEGER PRIMARY KEY, mode TEXT," + " title TEXT, model TEXT, created_at INTEGER, updated_at INTEGER);" + "CREATE TABLE messages (id INTEGER PRIMARY KEY, session_id INTEGER," + " role TEXT NOT NULL, content TEXT NOT NULL," + " prompt_tokens INTEGER, completion_tokens INTEGER, model TEXT," + " created_at INTEGER NOT NULL);" + "INSERT INTO sessions VALUES (1,'chat','old','m',0,0);" + "INSERT INTO messages VALUES (1,1,'assistant','older reply'," + "1200,340,'m',0);" + ) + conn.commit() + conn.close() + + history = db.History(path) + # The pre-migration row reads as unknown, never as an empty string + # or a fabricated zero. + old = history.messages(1)[0] + assert old["provider"] is None + assert old["usage_json"] is None + assert old["reported_cost_usd"] is None + + mid = history.add_message(1, "assistant", "") + history.update_message( + mid, "new reply", prompt_tokens=10, completion_tokens=5, + model="q:r", provider="q", usage_json='{"total_tokens":15}', + ) + fresh = history.messages(1)[1] + assert fresh["provider"] == "q" + assert fresh["usage_json"] == '{"total_tokens":15}' + assert fresh["reported_cost_usd"] is None + history.close() + print("ok usage column migration") + + def test_client_auth_header(): """A client with a key sends Bearer auth; one without sends no header.""" sent = {} @@ -1908,6 +2073,33 @@ def test_client_auth_header(): print("ok client authorization header") +def test_debug_log(): + """The diagnostic log is off unless $LLAMACHAT_DEBUG_LOG names a path.""" + with tempfile.TemporaryDirectory() as tmp: + path = Path(tmp) / "debug.log" + saved = os.environ.get("LLAMACHAT_DEBUG_LOG") + try: + # Unset -> a no-op that creates nothing to rotate. + os.environ.pop("LLAMACHAT_DEBUG_LOG", None) + backend._debug_log("must not be written") + assert not path.exists() + + # Set -> appends timestamped records. + os.environ["LLAMACHAT_DEBUG_LOG"] = str(path) + backend._debug_log("first") + backend._debug_log("second") + lines = path.read_text(encoding="utf-8").strip().splitlines() + assert len(lines) == 2, lines + assert lines[0].startswith("[") and "first" in lines[0] + assert "second" in lines[1] + finally: + if saved is None: + os.environ.pop("LLAMACHAT_DEBUG_LOG", None) + else: + os.environ["LLAMACHAT_DEBUG_LOG"] = saved + print("ok debug log") + + def test_stream_body_for_cloud(): """Cloud providers get max_tokens and thinking_budget; local does not.""" captured = {} @@ -2507,6 +2699,115 @@ def test_search_failure_paths(): print("ok search failure paths") +def test_reasoning_content_preserved(): + """A tool-call round replays its reasoning_content verbatim. + + DeepSeek V3.2+/V4 and GLM-4.7+ on SiliconFlow emit chain-of-thought as + reasoning_content and require it sent back unchanged on the assistant + tool-call message. Dropping it breaks their multi-step tool flow, which + surfaces as a turn that stops at the thinking with no answer. + """ + import urllib.request + + from llamachat import search + + client = backend.Client("http://x") + seen_messages = [] + + def scripted(model, messages, tools): + seen_messages.append([dict(m) for m in messages]) + if len(seen_messages) == 1: + yield ("reasoning", "I should check the current version.") + yield from _tool_round("latest kernel") + else: + yield ("content", "answered") + + client._stream_once = scripted + original = _with_urlopen(json.dumps({"results": []}).encode()) + try: + cfg = backend.SearchConfig( + enabled=True, url="http://searx", max_searches=1 + ) + list(client.stream_chat("m", [{"role": "user", "content": "hi"}], cfg)) + finally: + urllib.request.urlopen = original + + # The second request's assistant message must carry the thinking back. + second = seen_messages[1] + assistant = next(m for m in second if m["role"] == "assistant" and m.get("tool_calls")) + assert assistant["reasoning_content"] == "I should check the current version.", assistant + + # The reasoning still reaches the UI unchanged. + assert len(seen_messages) == 2, seen_messages + print("ok reasoning content preserved") + + +def test_tool_call_finish_with_usage(): + """finish_reason riding the usage chunk still triggers the search. + + DeepSeek's include_usage stream puts finish_reason: "tool_calls" on the + same SSE line as the usage block. The old single-event parser returned + that line as "usage" and never saw the finish, so the tool call was + dropped and the turn ended at the thinking with no search. + """ + import httpx + import urllib.request + + def sse(obj): + return "data: " + json.dumps(obj) + + lines = [ + sse({"choices": [{"delta": {"reasoning_content": "hmm"}}]}), + sse({"choices": [{"delta": {"tool_calls": [ + { + "index": 0, + "id": "call_1", + "type": "function", + "function": {"name": "web_search", "arguments": '{"query":"kernel"}'}, + } + ]}}]}), + sse({ + "choices": [{ + "index": 0, + "delta": {"content": "", "reasoning_content": None}, + "finish_reason": "tool_calls", + }], + "usage": {"prompt_tokens": 10, "completion_tokens": 5, "total_tokens": 15}, + }), + "data: [DONE]", + ] + + class _Stream: + status_code = 200 + + def __enter__(self): + return self + + def __exit__(self, *exc): + return False + + def iter_lines(self): + return iter(lines) + + def _fake_stream(method, url, json=None, timeout=None, headers=None): + return _Stream() + + client = backend.Client("http://x") + original = _with_urlopen(json.dumps({"results": []}).encode()) + try: + with _patched(httpx, "stream", _fake_stream): + cfg = backend.SearchConfig(enabled=True, url="http://searx", max_searches=1) + out = list(client.stream_chat("m", [{"role": "user", "content": "hi"}], cfg)) + finally: + urllib.request.urlopen = original + + kinds = [k for k, _ in out] + assert "search_start" in kinds, kinds + assert "search_done" in kinds, kinds + assert ("reasoning", "hmm") in out, out + print("ok tool call finish with usage") + + def test_search_storage(): import sqlite3 @@ -2877,10 +3178,15 @@ if __name__ == "__main__": test_model_ids_and_filtering() test_key_resolution() test_config_providers() + test_replays_reasoning() test_models_store() test_metadata_and_cost() test_token_column_migration() + test_usage_columns() + test_usage_accumulation() + test_usage_column_migration() test_client_auth_header() + test_debug_log() test_stream_body_for_cloud() test_multi_client() test_model_dialog_values() @@ -2890,6 +3196,8 @@ if __name__ == "__main__": test_tool_call_accumulation() test_search_loop_cap() test_search_failure_paths() + test_reasoning_content_preserved() + test_tool_call_finish_with_usage() test_search_storage() test_date_note() test_search_html() |
