diff options
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 92 |
1 files changed, 90 insertions, 2 deletions
@@ -33,6 +33,10 @@ persistent process and toggles like a scratchpad from a Hyprland keybind. any conversation can take a one-off custom prompt or none at all. The choice is stored per session, so reopening an old chat restores the prompt it was built with. +- **Web search** through a SearXNG instance, off by default. The model is + offered a `web_search` tool and calls it when a question needs current + information; the queries and their sources show as a collapsible block + above the reply. See [Web search](#web-search). - **Tray icon** for show/hide/quit, hosted by waybar's tray module. Not in this version: RAG or embedding search over history, multi-user @@ -132,6 +136,15 @@ default_prompt = "" attach_ctx_fraction = 0.5 chars_per_token = 3.5 + +# Web search. Both keys are required: a flag with no URL stays off. +search_enabled = false +search_url = "" + +search_results = 5 +search_snippet_chars = 300 +search_timeout = 10 +max_searches = 2 ``` `presets.ini` is read for two things the API does not report: which models @@ -242,7 +255,8 @@ llamachat.py launcher, venv shebang, symlink target llamachat/ __main__.py argument dispatch, daemon, tray, socket event loop config.py config.toml and presets.ini parsing - backend.py httpx streaming client, attachment loading + backend.py httpx streaming client, tool loop, attachment loading + search.py SearXNG queries, tool schema, result sanitising db.py SQLite schema, FTS5 search ui.py the window test_llamachat.py self-checks for everything except the GUI @@ -271,7 +285,7 @@ there is no second thread. ```sql sessions (id, mode, title, model, prompt_name, prompt_custom, created_at, updated_at) -messages (id, session_id, role, content, reasoning, created_at) +messages (id, session_id, role, content, reasoning, searches, created_at) attachments (id, message_id, path, kind, mime, size, sha256, thumb, truncated) messages_fts -- FTS5 external-content table over messages.content ``` @@ -285,6 +299,11 @@ reasons: the FTS index covers `content` only, so thinking text cannot bury real search hits, and only `content` is replayed when continuing a conversation, which is what these APIs expect. +`searches` holds a JSON array of what was looked up for that reply, one entry +per search with its query, results and any error, or NULL when nothing was +searched. Like `reasoning` it stays out of the FTS index, or snippets from +web pages would compete with messages the user actually wrote. + `prompt_name` records which system prompt a conversation was built with, and `prompt_custom` holds the text when that prompt is a one-off rather than a file. Storing the name rather than the resolved text means editing a preset @@ -331,6 +350,75 @@ a preset name, `"none"`, or empty for `default.md`. The choice is recorded per session, so reopening a chat restores the prompt it was built with rather than silently applying whatever is selected now. +### Web search + +Off unless both `search_enabled = true` and a `search_url` are set. A flag +with no URL behind it resolves to disabled at startup, so an upgrade never +starts making network requests on its own. + +The model decides when to search. It is offered an OpenAI-style `web_search` +function, and when it judges a question needs current information it calls +it; llamachat queries SearXNG's JSON API, hands back the results, and the +model answers from them. Nothing needs enabling on the llama-server side: +this is plain tool calling, not llama-server's `--tools` (which runs +server-side tools, none of which search the web) and not `--ui-mcp-proxy` +(a CORS shim that exists for browsers). + +The SearXNG instance must have its JSON output format enabled, which many +installs turn off by default. Without it, queries fail with a message saying +so. + +Searches show in the transcript as a collapsible block above the thinking +block, with a status line while one runs: + +``` +▸ searched: latest stable linux kernel (5 results) +▸ thinking (1441 chars, 38 lines) +Model: +... +``` + +Expanding it lists each result's title, link and snippet. The queries are +visible on purpose: when an answer is wrong it is usually the query that was +wrong, and without seeing it a bad search and a bad answer look identical. + +`max_searches` caps searches per turn, default 2. On the last round the tool +is withdrawn from the request, which forces the model to answer instead of +searching again. A local 9B will otherwise keep searching when it is unsure. + +Failures do not abort the turn. A timeout, a refused connection, a non-JSON +response or zero results all come back to the model as a tool result saying +what happened, so it answers with that knowledge rather than silently +falling back on stale training data. The block then reads +`▸ search failed: <query> — <reason>`. + +A search that returns nothing *because every engine failed* counts as a +failure rather than an empty result, and the message names the engines. On +a self-hosted SearXNG the usual way search stops working is engines getting +rate-limited or CAPTCHA'd one by one, and reporting that as "no results" +tells the model the web is empty, which invites exactly the confident wrong +answer this feature exists to prevent. + +When search is on, the current date is appended to the system prompt. A +model has no clock, so it falls back on its training cutoff and writes that +year into the search query itself, poisoning the results before they are +fetched. It is also told not to put a year in a query unless asked. This is +added even when `no system prompt` is selected, since the model has no clock +either way. + +**On trusting results.** Search snippets are attacker-influenced text going +into the model's context, in an app whose output you may paste into a +terminal. Structurally: only `title`, `url` and the snippet are kept and +everything else SearXNG returns is dropped, snippets are truncated to +`search_snippet_chars`, all result text is escaped when displayed, result +links are shown but never fetched automatically, and a reply that forges the +search block's internal URL scheme has it defused. The tool result is also +labelled untrusted for the model, which helps inconsistently. + +None of that stops a poisoned snippet from influencing what the model says. +If a top result asserts something false, a 9B may repeat it. That is why the +sources stay visible rather than being hidden behind the answer. + ### The context meter The bar in the top bar shows how much of the active model's context window |