aboutsummaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authorDanilo M. <danix@danix.xyz>2026-07-31 19:09:44 +0200
committerDanilo M. <danix@danix.xyz>2026-07-31 19:09:44 +0200
commit0e35416128e38f158af49935262ecbf8dbf585e6 (patch)
tree2af059613853634dc8968ad700e98651bb383514
parent283fc9ae0371f4d318a539f021a4b88e05fac190 (diff)
downloadllamachat-0e35416128e38f158af49935262ecbf8dbf585e6.tar.gz
llamachat-0e35416128e38f158af49935262ecbf8dbf585e6.zip
fix: cap searches at one per turn
A second tool call in the same turn, issued after the model has seen the first set of results, does not arrive as a tool_calls delta. It comes back as literal <tool_call><function=web_search> text inside reasoning_content, with finish_reason stop, no tool_calls and empty content. There is no structured call for the loop to act on, so the turn ends and the user is shown a blank reply. Reproduced 3/3 with Qwen3.5-9B through the router; the first call of a turn parses correctly every time, so this is specific to a call that follows a tool result. A retry with the tool withdrawn was tried and rejected: it produced an answer only some of the time and could surface the raw XML in the reply, which is worse than the blank it replaces. Capping at one search avoids reaching the broken state at all. Empty replies drop from 3/3 to about 1/3 and the XML no longer leaks, but the failure is not eliminated. Raise max_searches once the chat template parses follow-up calls. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
-rw-r--r--CHANGELOG.md6
-rw-r--r--README.md12
-rw-r--r--docs/superpowers/specs/2026-07-31-web-search-design.md16
-rw-r--r--llamachat/config.py11
-rwxr-xr-xtest_llamachat.py6
5 files changed, 47 insertions, 4 deletions
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 7b3aad6..ddfc863 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -22,7 +22,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Six configuration keys: `search_enabled`, `search_url`, `search_results`,
`search_snippet_chars`, `search_timeout` and `max_searches`. Searches are
capped per turn, after which the tool is withdrawn and the model must
- answer.
+ answer. The cap defaults to 1: a second tool call, issued once the model
+ has seen the first results, comes back as literal `<tool_call>` text
+ rather than a structured call, and a turn that ends on one leaves the
+ reply empty. Raise it when the model and llama.cpp build handle
+ follow-up calls.
- Searches are stored per message in a new `searches` column, so reopening a
conversation still shows what was looked up. They are deliberately kept
out of the full-text index, or web text nobody wrote would compete with
diff --git a/README.md b/README.md
index 546fa1b..9c0807f 100644
--- a/README.md
+++ b/README.md
@@ -144,7 +144,7 @@ search_url = ""
search_results = 5
search_snippet_chars = 300
search_timeout = 10
-max_searches = 2
+max_searches = 1
```
`presets.ini` is read for two things the API does not report: which models
@@ -382,10 +382,18 @@ Expanding it lists each result's title, link and snippet. The queries are
visible on purpose: when an answer is wrong it is usually the query that was
wrong, and without seeing it a bad search and a bad answer look identical.
-`max_searches` caps searches per turn, default 2. On the last round the tool
+`max_searches` caps searches per turn, default 1. On the last round the tool
is withdrawn from the request, which forces the model to answer instead of
searching again. A local 9B will otherwise keep searching when it is unsure.
+The default is 1 rather than 2 because of an upstream limitation. The first
+tool call of a turn arrives as a proper `tool_calls` delta, but a second one,
+issued after the model has seen the first set of results, comes back as
+literal `<tool_call><function=web_search>` text inside the thinking instead.
+There is no structured call to act on, so the turn ends with an empty reply.
+Observed with Qwen3.5-9B through llama.cpp's router; raise the cap if your
+model and build handle follow-up calls properly.
+
Failures do not abort the turn. A timeout, a refused connection, a non-JSON
response or zero results all come back to the model as a tool result saying
what happened, so it answers with that knowledge rather than silently
diff --git a/docs/superpowers/specs/2026-07-31-web-search-design.md b/docs/superpowers/specs/2026-07-31-web-search-design.md
index 7e14b2d..46ebc91 100644
--- a/docs/superpowers/specs/2026-07-31-web-search-design.md
+++ b/docs/superpowers/specs/2026-07-31-web-search-design.md
@@ -235,6 +235,22 @@ at the end, outside the suite.
- **Citation formatting in replies** — depends on 9B instruction-following;
the search block already shows sources
+## Implementation note: the cap shipped as 1, not 2
+
+The table above chose a cap of two searches per turn. It shipped as one.
+
+The first tool call of a turn arrives as a well-formed `tool_calls` delta,
+exactly as designed. A *second* call, issued after the model has seen the
+first set of results, comes back as literal
+`<tool_call><function=web_search>` text inside `reasoning_content`, with
+`finish_reason: stop`, no `tool_calls`, and empty `content`. There is no
+structured call for the loop to act on, so the turn ends with a blank reply.
+
+Reproduced 3/3 with Qwen3.5-9B through the router. A retry with the tool
+withdrawn was tried and rejected: it recovered an answer only some of the
+time and could surface the raw XML to the user, which is worse than the
+blank it replaces. The cap is 1 until the template parses follow-up calls.
+
## Notes
`reasoning-budget = 1024` in the active `Qwen3.5-9B` preset is tight for tool
diff --git a/llamachat/config.py b/llamachat/config.py
index d7682f0..692dd79 100644
--- a/llamachat/config.py
+++ b/llamachat/config.py
@@ -44,7 +44,11 @@ DEFAULTS = {
"search_results": 5,
"search_snippet_chars": 300,
"search_timeout": 10,
- "max_searches": 2,
+ # One search per turn. A second call is emitted by the model as literal
+ # <tool_call> XML rather than a tool_calls delta, which this code cannot
+ # act on and which leaves the reply empty. Raise it once the template
+ # parses follow-up calls.
+ "max_searches": 1,
}
# The global prompt lives here; every other .md beside it is a named preset.
@@ -185,6 +189,11 @@ def write_default(path: Path = CONFIG_PATH) -> Path:
'\n'
'# Results per query, characters kept from each snippet, seconds\n'
'# before a query is abandoned, and searches allowed per turn.\n'
+ '#\n'
+ '# max_searches is 1 because a second tool call comes back as\n'
+ '# literal <tool_call> XML instead of a structured call, which\n'
+ '# ends the turn with an empty reply. Raise it if your model and\n'
+ '# llama.cpp build handle follow-up calls properly.\n'
f'search_results = {DEFAULTS["search_results"]}\n'
f'search_snippet_chars = {DEFAULTS["search_snippet_chars"]}\n'
f'search_timeout = {DEFAULTS["search_timeout"]}\n'
diff --git a/test_llamachat.py b/test_llamachat.py
index 936a18d..a766382 100755
--- a/test_llamachat.py
+++ b/test_llamachat.py
@@ -967,6 +967,12 @@ def test_search_tool_schema():
assert out == [("content", "hi")]
assert sent == [None], sent
+ # One search per turn by default: a follow-up call arrives as literal
+ # <tool_call> XML rather than a structured call, and a turn that ends on
+ # one shows the user an empty reply.
+ assert config.DEFAULTS["max_searches"] == 1
+ assert config.load(Path("/nonexistent/config.toml")).max_searches == 1
+
# A configured-but-urlless setup resolves to disabled at config load.
with tempfile.TemporaryDirectory() as tmp:
path = Path(tmp) / "config.toml"