aboutsummaryrefslogtreecommitdiffstats
path: root/README.md
diff options
context:
space:
mode:
authorDanilo M. <danix@danix.xyz>2026-08-01 20:03:56 +0200
committerDanilo M. <danix@danix.xyz>2026-08-01 20:03:56 +0200
commit2fae45408489594d009215c82c2ca9acdc00e394 (patch)
treeed5587690429b360e898433eadc85e00158760f5 /README.md
parentfe335db0cdc72cc190c290479d24c6469e2c07ab (diff)
downloadllamachat-2fae45408489594d009215c82c2ca9acdc00e394.tar.gz
llamachat-2fae45408489594d009215c82c2ca9acdc00e394.zip
feat: model-written titles, fix empty replies after search
Titles: once a reply lands in a session still wearing its placeholder title, the exchange goes back to the model in a short side request asking for six words or fewer. Keyed on the placeholder rather than the turn number, so an empty first reply does not forfeit titling for the session and a one-shot window is titled from the question that was answered. Thinking is disabled for that request, or a reasoning model spends the whole budget thinking and returns nothing. Empty replies: the final search round now says so in the tool result. Withdrawing the tool schema is invisible to the model, which asks for another search regardless; the request then surfaces as literal <tool_call> text or vanishes into the thinking block, leaving the reply empty either way. The note rides on the tool result because Qwen3.5's template rejects a trailing system message outright. max_searches now defaults to 2. The other half of that bug was llama-server's quantized KV cache: with cache-type-k/v = q8_0 three of five turns broke, and f16 answered six of six. Measured against a live router, not mocked.
Diffstat (limited to 'README.md')
-rw-r--r--README.md47
1 files changed, 31 insertions, 16 deletions
diff --git a/README.md b/README.md
index 8071ec6..ada0d5a 100644
--- a/README.md
+++ b/README.md
@@ -17,6 +17,9 @@ persistent process and toggles like a scratchpad from a Hyprland keybind.
vision model, and dropping one on a text model offers to switch.
- **History** in SQLite with FTS5 full-text search. Past chat sessions reopen
and continue with their context intact; one-shot entries reopen read-only.
+ Sessions are named by the model itself: once the first reply lands, it is
+ asked to title the exchange in a few words, which replaces the opening
+ words of the question the entry was created with.
- **Streaming replies** rendered token by token, formatted as markdown:
headings, bold and italic, bullet and numbered lists, tables, inline code
and tinted fenced code blocks. What you type is shown exactly as typed, so
@@ -144,7 +147,7 @@ search_url = ""
search_results = 5
search_snippet_chars = 300
search_timeout = 10
-max_searches = 1
+max_searches = 2
```
`presets.ini` is read for two things the API does not report: which models
@@ -382,21 +385,33 @@ Expanding it lists each result's title, link and snippet. The queries are
visible on purpose: when an answer is wrong it is usually the query that was
wrong, and without seeing it a bad search and a bad answer look identical.
-`max_searches` caps searches per turn, default 1. On the last round the tool
-is withdrawn from the request, which forces the model to answer instead of
-searching again. A local 9B will otherwise keep searching when it is unsure.
-
-The default is 1 rather than 2 because of an upstream limitation. The first
-tool call of a turn arrives as a proper `tool_calls` delta, but a second one,
-issued after the model has seen the first set of results, often comes back as
-literal `<tool_call><function=web_search>` text inside the thinking instead.
-There is no structured call to act on, so the turn ends with an empty reply.
-
-Observed with Qwen3.5-9B through llama.cpp's router. Newer builds parse
-follow-up calls some of the time (1 in 5 on b10208) rather than never, so
-this may be worth retesting; raise the cap if your model and build handle
-them reliably. Enabling more search engines helps more than raising the cap,
-since a first round that returns plenty removes the reason to search twice.
+`max_searches` caps searches per turn, default 2. On the last round the tool
+is withdrawn from the request and the tool result says so in words, which is
+what forces the model to answer instead of searching again. A local 9B will
+otherwise keep searching when it is unsure.
+
+Withdrawing the schema on its own is not enough, because the model cannot
+see a schema disappear. It asks for another search regardless, and that
+request surfaces either as literal `<tool_call><function=web_search>` text
+in the reply or buried in the thinking block, leaving the reply itself
+empty. Saying "this was your last search" in the tool result is what fixes
+it. The note has to ride on the tool result rather than a trailing system
+message: Qwen3.5's chat template rejects those outright, raising `System
+message must be at the beginning`.
+
+If replies still come back empty after a search, check the server's KV
+cache. llama.cpp's own function-calling documentation warns that quantized
+KV degrades tool calling, and it does: with `cache-type-k`/`cache-type-v`
+set to `q8_0`, three of five turns on one question came back broken, and
+dropping to the `f16` default made the same question answer six times out
+of six.
+
+Turns that search once are reliable. Turns that genuinely need two are not
+quite: on a question forcing two rounds, four and five of six succeeded
+across two runs, the rest coming back empty. Lower `max_searches` to 1 if
+you would rather never see an empty reply, or if you keep the quantized
+cache. Enabling more search engines helps either way, since a first round
+that returns plenty removes the reason to search twice.
Failures do not abort the turn. A timeout, a refused connection, a non-JSON
response or zero results all come back to the model as a tool result saying