diff options
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 47 |
1 files changed, 31 insertions, 16 deletions
@@ -17,6 +17,9 @@ persistent process and toggles like a scratchpad from a Hyprland keybind. vision model, and dropping one on a text model offers to switch. - **History** in SQLite with FTS5 full-text search. Past chat sessions reopen and continue with their context intact; one-shot entries reopen read-only. + Sessions are named by the model itself: once the first reply lands, it is + asked to title the exchange in a few words, which replaces the opening + words of the question the entry was created with. - **Streaming replies** rendered token by token, formatted as markdown: headings, bold and italic, bullet and numbered lists, tables, inline code and tinted fenced code blocks. What you type is shown exactly as typed, so @@ -144,7 +147,7 @@ search_url = "" search_results = 5 search_snippet_chars = 300 search_timeout = 10 -max_searches = 1 +max_searches = 2 ``` `presets.ini` is read for two things the API does not report: which models @@ -382,21 +385,33 @@ Expanding it lists each result's title, link and snippet. The queries are visible on purpose: when an answer is wrong it is usually the query that was wrong, and without seeing it a bad search and a bad answer look identical. -`max_searches` caps searches per turn, default 1. On the last round the tool -is withdrawn from the request, which forces the model to answer instead of -searching again. A local 9B will otherwise keep searching when it is unsure. - -The default is 1 rather than 2 because of an upstream limitation. The first -tool call of a turn arrives as a proper `tool_calls` delta, but a second one, -issued after the model has seen the first set of results, often comes back as -literal `<tool_call><function=web_search>` text inside the thinking instead. -There is no structured call to act on, so the turn ends with an empty reply. - -Observed with Qwen3.5-9B through llama.cpp's router. Newer builds parse -follow-up calls some of the time (1 in 5 on b10208) rather than never, so -this may be worth retesting; raise the cap if your model and build handle -them reliably. Enabling more search engines helps more than raising the cap, -since a first round that returns plenty removes the reason to search twice. +`max_searches` caps searches per turn, default 2. On the last round the tool +is withdrawn from the request and the tool result says so in words, which is +what forces the model to answer instead of searching again. A local 9B will +otherwise keep searching when it is unsure. + +Withdrawing the schema on its own is not enough, because the model cannot +see a schema disappear. It asks for another search regardless, and that +request surfaces either as literal `<tool_call><function=web_search>` text +in the reply or buried in the thinking block, leaving the reply itself +empty. Saying "this was your last search" in the tool result is what fixes +it. The note has to ride on the tool result rather than a trailing system +message: Qwen3.5's chat template rejects those outright, raising `System +message must be at the beginning`. + +If replies still come back empty after a search, check the server's KV +cache. llama.cpp's own function-calling documentation warns that quantized +KV degrades tool calling, and it does: with `cache-type-k`/`cache-type-v` +set to `q8_0`, three of five turns on one question came back broken, and +dropping to the `f16` default made the same question answer six times out +of six. + +Turns that search once are reliable. Turns that genuinely need two are not +quite: on a question forcing two rounds, four and five of six succeeded +across two runs, the rest coming back empty. Lower `max_searches` to 1 if +you would rather never see an empty reply, or if you keep the quantized +cache. Enabling more search engines helps either way, since a first round +that returns plenty removes the reason to search twice. Failures do not abort the turn. A timeout, a refused connection, a non-JSON response or zero results all come back to the model as a tool result saying |