diff options
Diffstat (limited to 'docs/superpowers/specs')
| -rw-r--r-- | docs/superpowers/specs/2026-08-12-tagging-rules-design.md | 341 |
1 files changed, 341 insertions, 0 deletions
diff --git a/docs/superpowers/specs/2026-08-12-tagging-rules-design.md b/docs/superpowers/specs/2026-08-12-tagging-rules-design.md new file mode 100644 index 0000000..1d32f02 --- /dev/null +++ b/docs/superpowers/specs/2026-08-12-tagging-rules-design.md @@ -0,0 +1,341 @@ +# Shared tagging rules: design + +Backlog item 44, "No way to manage the filters applied at sync time". + +**Status:** design approved 2026-08-12, not implemented. + +## The problem this solves + +The user asked for a way to "manage filters to be applied when syncing (view +existing, edit, delete, create new, copy as new, dry-run)". The backlog recorded +the item as unspecified for a week because nothing in qtmaildir applies rules at +sync time, and the item could not be planned until it was known whether such +rules existed anywhere. + +They do. They live in the notmuch `post-new` hook inside the user's Maildir, as +a sequence of hand-written `notmuch tag` commands, each scoped to `tag:new` so +it applies to newly indexed mail only. The hook is tag-only by design: it never +removes `inbox` or `unread`, so nothing is archived or marked read unattended. + +Those rules are invisible from every tool. They are also where a substantial +amount of hard-won reasoning lives, in comments recording which senders were +deliberately excluded from a rule and why. That reasoning is readable only by +opening the hook in an editor, and it is exactly the context a person needs when +deciding whether a rule is still correct. + +## What is being built + +A rule store both tools read, in a format neither owns. + +``` + ~/.config/mailrules/rules.json + | + +---------------+---------------+--------------+ + | | | | + post-new qtmaildir qtmaildir mailctl rules + (mailctl) TagRules dialog list/show/ + | read+write dry-run dry-run + | | | | + scope: atomic via read-only + tag:new QSaveFile NotmuchWorker + | (generation- + notmuch tag stamped) + per rule + | + -new consumer + (hardcoded, + only if all + rules ran) +``` + +The constraint that shaped this: qtmaildir is the main consumer of the mail +system, but mailctl must keep working through it. A format with one +implementation is not neutral, it is qtmaildir's format that mailctl is invited +to parse. Both readers are therefore written together, and the format carries an +explicit provision for fields one tool does not understand. + +## The file + +`$XDG_CONFIG_HOME/mailrules/rules.json`, falling back to +`~/.config/mailrules/rules.json`. Neither tool hardcodes a home directory, and +the location is under neither project's own config directory. + +```json +{ + "version": 1, + "rules": [ + { + "id": "account-work", + "stage": 10, + "enabled": true, + "add": ["account-work"], + "remove": [], + "query": "path:\"work-account/**\"", + "note": "Account tags use the maildir spelling, not the account keys another tool shows. Must run before topic rules." + }, + { + "id": "notify-forge", + "stage": 50, + "enabled": true, + "add": ["notify/forge"], + "remove": [], + "query": "from:notifications@example.com", + "note": "All repositories, not one project." + } + ] +} +``` + +JSON, and not the INI the item originally named. Both tools parse JSON with no +new dependency: `QJsonDocument` in Qt, `json` in the Python standard library, +and mailctl is deliberately stdlib-only. INI was rejected on a concrete hazard +rather than taste: tag names in use contain `/` (`notify/forge`, +`mailing-list/*`, `shopping/*`), and `QSettings` treats `/` in a section name as +a group separator. `CLAUDE.md` already records that trap costing this project +once, over the `[general]` section. + +The cost of JSON is that the file cannot carry loose comments. This is +acceptable because every comment in the current hook is attached to a specific +rule, and `note` gives it a home that a UI can display. The reasoning stops +being visible only to someone reading shell. + +### Fields + +| Field | Meaning | +|---|---| +| `version` | Format version, currently `1`. A reader finding a higher number refuses the file rather than guessing at semantics. | +| `id` | Stable unique handle, `[a-z0-9-]`. What a UI selects and a diff tracks. | +| `stage` | Integer, ascending. Ties broken by array order. | +| `enabled` | A rule switched off without losing its note. | +| `add` / `remove` | Tags, as separate arrays. | +| `query` | notmuch query. Carries no scope and no `tag:new`. | +| `note` | Why this rule is shaped the way it is. | + +**`id` is never the tag name.** Tags contain `/`, and a tag can be renamed while +the rule stays the same rule. + +**`add` and `remove` are arrays, not `+tag`/`-tag` strings.** Every current rule +only adds, but the format should not need a version bump the first time a rule +wants a removal, and parsing sigils off strings invites the class of bug where a +tag legitimately beginning with a sigil character is mangled. + +**Unknown keys are preserved across a write by either tool.** A reader stashes +every field it does not recognise and a writer merges them back. Without this, +one qtmaildir save silently strips whatever a newer mailctl wrote, and the file +is neutral in name only. + +## Scope belongs to the runner, not the rule + +A rule is a `(add, remove, query, stage)` tuple that knows nothing about which +mail it applies to. Whoever runs it supplies the scope. + +| Runner | Scope | Effect | +|---|---|---| +| `post-new` | `tag:new` | Tags newly indexed mail. | +| Dry run | none | Counts against the whole corpus. | +| Backfill (future) | none | Applies to the whole corpus. | +| A future timer or hook | its own | Not designed here. | + +This is why stored queries omit `tag:new`. A query that carried its own scope +would have to be stripped back down for a dry run to be useful, and a +hand-written rule that forgot the scope would silently retag the entire archive. + +It also settles where the `tag:new` consumer lives. `notmuch tag -new -- tag:new` +is not a tagging rule, it is the `post-new` runner's own cleanup, and it is +hardcoded there rather than stored as data. It cannot be deleted, disabled or +reordered by either tool because it is not a rule. The failure mode if it went +missing justifies putting it out of reach: `tag:new` would accumulate, and every +rule would begin matching the whole backlog instead of new arrivals. + +Rules genuinely untied to `tag:new` already exist as manual work today: +retroactively applying a newly written rule, and re-running a corrected rule +over history after a mistake. Both are the same rule at a different scope, which +is what this split makes expressible. + +## The runner + +Today's hook is a sequence of hand-written `notmuch tag` lines. It becomes a +loop over `rules.json`. + +**Python, not shell.** The hook must parse JSON. Shell would mean either a new +`jq` dependency on a path that runs every ten minutes, or hand-parsing JSON in +`sh`. Python 3 is already required by mailctl and needs no imports beyond the +standard library here. + +**It lives in the mailctl repository, installed to `~/bin/`.** The hook is a +mail-organization concern and mailctl is the mail-organization tool. Note the +asymmetry with `mailsync.sh`, which lives in qtmaildir precisely because +qtmaildir runs it as a subprocess and depends on its behaviour; here the +relationship is reversed, since qtmaildir never invokes the hook and `notmuch +new` does. + +``` +mailctl repo + mailrules.py load/save/validate/stage-order, unknown-key preservation + post-new reads rules.json, applies with tag:new scope, then -new + mailctl.py mailctl rules list|show|dry-run + +qtmaildir repo + src/tagrules.* the same format, read/write/validate in C++ + (dialog) view, edit, create, delete, copy-as-new, dry-run +``` + +### Algorithm, and the properties that must survive any edit + +1. Load `rules.json`. **On any error, missing file, bad JSON or failed + validation, log to stderr and exit non-zero WITHOUT running the `-new` + consumer.** +2. Sort enabled rules by `stage`, then by array position. +3. For each rule, run + `notmuch tag <+add...> <-remove...> -- 'tag:new and (<query>)'`. +4. **Only after every rule succeeded**, run `notmuch tag -new -- tag:new`. + +**Step 1 is the critical safety property.** If the consumer runs while the rules +did not, `tag:new` is cleared from mail that was never tagged, and that mail can +never be tagged by these rules again. The failure is silent, permanent, and +invisible until someone notices a gap months later. A rules file that fails to +load must leave `tag:new` in place so the next successful run catches up. + +**The query is parenthesised in step 3, and this is not cosmetic.** Several +current rules are a disjunction of senders. Without the parentheses, +`tag:new and a or b` binds as `(tag:new and a) or b`, and the rule matches every +message in the corpus that satisfies `b` rather than only new arrivals. + +**Step 4 makes the run idempotent.** A failure partway through leaves `tag:new` +set, so re-running is safe and completes the work. This is also what makes the +migration verifiable: old and new hooks can be run against the same `tag:new` +set and compared. + +## qtmaildir + +**`TagRules` (`src/tagrules.h`, `src/tagrules.cpp`)**, a plain value type with +no widget dependency, following `Config` and `KeyMap`: it parses, validates, +collects warnings rather than throwing, and is unit-testable without a UI. +`QJsonDocument` to parse and serialize, `QSaveFile` to write. + +```cpp +struct TagRule { + QString id, query, note; + QStringList add, remove; + int stage = 50; + bool enabled = true; + QJsonObject unknown; // fields this version does not know, preserved on save +}; +``` + +The `unknown` member is the neutrality guarantee made concrete. + +**Validation degrades, it does not refuse.** Following `Config`'s existing +pattern: a duplicate id, an empty query, or a rule with neither `add` nor +`remove` drops that rule and records a warning the dialog shows. A file that +will not parse leaves the list empty with one warning. qtmaildir must never fail +to open because of this file. + +**Dry-run runs on the worker.** `NotmuchWorker` owns the only +`notmuch_database_t*` and notmuch permits one open handle per process, so +counting cannot happen on the UI thread even if it were fast. It is not fast: +item 74 measured a 4444-thread query at 5.7 seconds against a cold page cache, +and a dry run counts every rule. The counting slot is generation-stamped like +every other query so a superseded result is discarded. + +Each rule shows two numbers: + +- **Corpus count**, what the rule matches across all mail. This is the number + that answers "is this rule still doing anything" and "would this rule bury + real correspondence", which is what the comments in the current hook spend + most of their words reasoning about. +- **Pending count**, what it would tag on the next sync. Usually zero between + syncs. + +**The dialog** lists rules in stage order, each showing its tags, query, note, +enabled state and counts. Edit, create, delete and copy-as-new operate on the +list; Save writes atomically. Rules are a property of the mail system rather +than of an account, so the account dropdown does not scope them. + +## mailctl + +Read-only in this version: + +``` +mailctl rules list # id, stage, tags, enabled, note +mailctl rules show <id> # one rule, full query and note +mailctl rules dry-run [<id>] # corpus counts, all rules or one +``` + +**No `rules edit`, `add` or `delete`.** mailctl's safety model is that reads are +free and mutations are gated, and a rule edit is a mutation whose blast radius is +every future sync. What that gate should be is a real design question and it is +not needed yet: qtmaildir has the editor, and the file has a text editor. Writing +is fully designed and implemented in `mailrules.py`, because the hook's library +and qtmaildir must agree on atomic-write and unknown-key semantics, but no CLI +surface exposes it. + +`dry-run` reuses the existing `run_notmuch` and counting paths. **Its numbers +must equal qtmaildir's for the same rule**, and checking that once by hand after +migration is the cheap cross-check that the two parsers agree. + +## Concurrency + +Atomic writes, no locking. Every writer does write-to-temp then `rename()`, +which is atomic within a filesystem, so a reader always sees a complete file, +old or new. A truncated read by the hook is therefore impossible, which is the +failure that would actually hurt. + +Last writer wins on a genuine collision, silently. Accepted: this is a +single-user setup, agents run only when asked, and simultaneous edits from two +tools are rare enough to be corrected by eye. Locking was rejected because a +stale lock would block the hook, which is the one thing that must not be +blocked. + +## Migration + +The existing rules are converted once, preserving each comment as the rule's +`note`. + +**The live hook is not touched until the user has reviewed the conversion.** The +new hook is staged alongside as `post-new.new` and swapped by the user. + +**Equivalence is proved, not assumed.** For each rule, `notmuch count` the old +query and the new `tag:new and (query)` construction against the full corpus, +unscoped, and compare rule by rule. Identical counts are the evidence that the +conversion changed nothing. + +The conversion table names real senders and is therefore built and kept locally, +outside version control, per the project rule against personal details in +committed documentation. Every example in this document uses placeholders. + +## Testing + +**qtmaildir, `tests/test_tagrules.cpp`:** parse; round-trip preserving unknown +fields; each validation warning; stage ordering including ties resolved by array +position; and JSON escaping of queries containing quotes, which is not +hypothetical since path-scoped queries are written `path:"account/**"`. + +**mailctl:** plain-assert tests for `mailrules.py` matching the existing +`test_mailctl.py` style, covering the same load, save and ordering semantics. + +**Cross-tool:** the dry-run counts from both tools compared once by hand after +migration. + +The risk here is in the file format and the runner's failure modes, not in +painting. `CLAUDE.md`'s warnings about rendering probes apply to the dialog if +it grows custom painting, which this design does not call for. + +## Out of scope for v1 + +**Backfill: applying a rule to existing mail.** The counts are shown; the button +is not built. This is wanted and is the natural next step, since "apply this new +rule to the backlog" is currently a hand-run `notmuch tag`. + +Two things must be settled before it lands. A rule that is safe against arrivals +is not automatically safe against the whole corpus: a rule removing `inbox`, run +unscoped, would archive years of mail in one action. And `CLAUDE.md` currently +states that this project has no destructive-action confirmation and uses undo +instead, which does not cover a bulk tag write over thousands of messages. **The +user has stated that rule is due for revision**; backfill is the change that +forces it. + +**Rule edits from mailctl**, pending the gating decision above. + +**`post-insert` and timer-driven rules.** The scope split makes them +expressible, and nothing is designed for them here. |