diff options
| -rw-r--r-- | CHANGELOG.md | 6 | ||||
| -rw-r--r-- | README.md | 12 | ||||
| -rw-r--r-- | docs/superpowers/specs/2026-07-31-web-search-design.md | 16 | ||||
| -rw-r--r-- | llamachat/config.py | 11 | ||||
| -rwxr-xr-x | test_llamachat.py | 6 |
5 files changed, 47 insertions, 4 deletions
diff --git a/CHANGELOG.md b/CHANGELOG.md index 7b3aad6..ddfc863 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -22,7 +22,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Six configuration keys: `search_enabled`, `search_url`, `search_results`, `search_snippet_chars`, `search_timeout` and `max_searches`. Searches are capped per turn, after which the tool is withdrawn and the model must - answer. + answer. The cap defaults to 1: a second tool call, issued once the model + has seen the first results, comes back as literal `<tool_call>` text + rather than a structured call, and a turn that ends on one leaves the + reply empty. Raise it when the model and llama.cpp build handle + follow-up calls. - Searches are stored per message in a new `searches` column, so reopening a conversation still shows what was looked up. They are deliberately kept out of the full-text index, or web text nobody wrote would compete with @@ -144,7 +144,7 @@ search_url = "" search_results = 5 search_snippet_chars = 300 search_timeout = 10 -max_searches = 2 +max_searches = 1 ``` `presets.ini` is read for two things the API does not report: which models @@ -382,10 +382,18 @@ Expanding it lists each result's title, link and snippet. The queries are visible on purpose: when an answer is wrong it is usually the query that was wrong, and without seeing it a bad search and a bad answer look identical. -`max_searches` caps searches per turn, default 2. On the last round the tool +`max_searches` caps searches per turn, default 1. On the last round the tool is withdrawn from the request, which forces the model to answer instead of searching again. A local 9B will otherwise keep searching when it is unsure. +The default is 1 rather than 2 because of an upstream limitation. The first +tool call of a turn arrives as a proper `tool_calls` delta, but a second one, +issued after the model has seen the first set of results, comes back as +literal `<tool_call><function=web_search>` text inside the thinking instead. +There is no structured call to act on, so the turn ends with an empty reply. +Observed with Qwen3.5-9B through llama.cpp's router; raise the cap if your +model and build handle follow-up calls properly. + Failures do not abort the turn. A timeout, a refused connection, a non-JSON response or zero results all come back to the model as a tool result saying what happened, so it answers with that knowledge rather than silently diff --git a/docs/superpowers/specs/2026-07-31-web-search-design.md b/docs/superpowers/specs/2026-07-31-web-search-design.md index 7e14b2d..46ebc91 100644 --- a/docs/superpowers/specs/2026-07-31-web-search-design.md +++ b/docs/superpowers/specs/2026-07-31-web-search-design.md @@ -235,6 +235,22 @@ at the end, outside the suite. - **Citation formatting in replies** — depends on 9B instruction-following; the search block already shows sources +## Implementation note: the cap shipped as 1, not 2 + +The table above chose a cap of two searches per turn. It shipped as one. + +The first tool call of a turn arrives as a well-formed `tool_calls` delta, +exactly as designed. A *second* call, issued after the model has seen the +first set of results, comes back as literal +`<tool_call><function=web_search>` text inside `reasoning_content`, with +`finish_reason: stop`, no `tool_calls`, and empty `content`. There is no +structured call for the loop to act on, so the turn ends with a blank reply. + +Reproduced 3/3 with Qwen3.5-9B through the router. A retry with the tool +withdrawn was tried and rejected: it recovered an answer only some of the +time and could surface the raw XML to the user, which is worse than the +blank it replaces. The cap is 1 until the template parses follow-up calls. + ## Notes `reasoning-budget = 1024` in the active `Qwen3.5-9B` preset is tight for tool diff --git a/llamachat/config.py b/llamachat/config.py index d7682f0..692dd79 100644 --- a/llamachat/config.py +++ b/llamachat/config.py @@ -44,7 +44,11 @@ DEFAULTS = { "search_results": 5, "search_snippet_chars": 300, "search_timeout": 10, - "max_searches": 2, + # One search per turn. A second call is emitted by the model as literal + # <tool_call> XML rather than a tool_calls delta, which this code cannot + # act on and which leaves the reply empty. Raise it once the template + # parses follow-up calls. + "max_searches": 1, } # The global prompt lives here; every other .md beside it is a named preset. @@ -185,6 +189,11 @@ def write_default(path: Path = CONFIG_PATH) -> Path: '\n' '# Results per query, characters kept from each snippet, seconds\n' '# before a query is abandoned, and searches allowed per turn.\n' + '#\n' + '# max_searches is 1 because a second tool call comes back as\n' + '# literal <tool_call> XML instead of a structured call, which\n' + '# ends the turn with an empty reply. Raise it if your model and\n' + '# llama.cpp build handle follow-up calls properly.\n' f'search_results = {DEFAULTS["search_results"]}\n' f'search_snippet_chars = {DEFAULTS["search_snippet_chars"]}\n' f'search_timeout = {DEFAULTS["search_timeout"]}\n' diff --git a/test_llamachat.py b/test_llamachat.py index 936a18d..a766382 100755 --- a/test_llamachat.py +++ b/test_llamachat.py @@ -967,6 +967,12 @@ def test_search_tool_schema(): assert out == [("content", "hi")] assert sent == [None], sent + # One search per turn by default: a follow-up call arrives as literal + # <tool_call> XML rather than a structured call, and a turn that ends on + # one shows the user an empty reply. + assert config.DEFAULTS["max_searches"] == 1 + assert config.load(Path("/nonexistent/config.toml")).max_searches == 1 + # A configured-but-urlless setup resolves to disabled at config load. with tempfile.TemporaryDirectory() as tmp: path = Path(tmp) / "config.toml" |