aboutsummaryrefslogtreecommitdiffstats
path: root/docs/superpowers/specs/2026-08-29-sync-status-file-design.md
diff options
context:
space:
mode:
authorDanilo M. <danix@danix.xyz>2026-08-29 11:25:02 +0200
committerDanilo M. <danix@danix.xyz>2026-08-29 11:25:02 +0200
commit3fd999907ae8344f76ee4e5be1ac278a26f452ca (patch)
tree17c01d44daee0147ecaca6cccd393471e6379cf1 /docs/superpowers/specs/2026-08-29-sync-status-file-design.md
parent8c78dd139a77e896c72b7fc8b799af3c0df34344 (diff)
downloadqtmaildir-3fd999907ae8344f76ee4e5be1ac278a26f452ca.tar.gz
qtmaildir-3fd999907ae8344f76ee4e5be1ac278a26f452ca.zip
feat: have the sync script report what it did
Item 174, and half of item 125. The premise was corrected before any code. The note asks for an external `notmuch new` to clear the pending count; it must not. That count means tag mutations not yet known to have reached the MAIL STORE, which is the server: an edit is in notmuch the moment it is made, and what is outstanding is mbsync pushing the renamed Maildir files. `notmuch new` re-indexes local files and pushes nothing, so clearing on it would tell the user their work was safe to quit on while it was still local. The entry's own proposal to watch notmuch_database_get_revision() was rejected for the same reason: a revision moves when mail ARRIVES too, and in neither case does it say anything about the server. What was actually wrong was the reporting channel. The application inferred a finished run from an inode in /proc/locks and from grepping the log for its RUN END banner, which made a human-readable line into wire format and could not say WHICH channels a run carried. The local sync path has always narrowed its clear to the accounts it carried; the external path could not, and cleared everything, so an edit to an account a run never touched was reported as delivered. So the script reports instead of leaving evidence to be inferred. It writes ~/.local/state/qtmaildir/syncstatus.json atomically at the end of every run, including a skip, naming the channels, both exit statuses and a state of ok, failed or skipped. MailSync::readStatus() reads it, MainWindow prefers it over the log banner and narrows the clear through Account::syncChannel(). A skipped run clears nothing, which is item 125's first half: the application can now see that a run happened and carried nothing. The log banner and lastRunOutcome() stay as the fallback for a missing file, which is what a first run after upgrading looks like. This is the user's own framing of the scope: the script was written for another system and adapted, and is now qtmaildir's only consumer, so it serves the application rather than the reverse. Two facts made it safe to act on: their crontab runs mailsync.sh and nothing else touches mail, and ~/bin/mailsync.sh is a symlink into this repo, so an edit is live on the next tick. Two bugs found while wiring it in, both recorded in the closed item. A test read the developer's real sync state, twice: a [sync] section naming only `log` leaves syncStatus() defaulting to the real file, so two tests asserting that a FAILED run leaves the count alone read the last real cron run, found ok, and cleared. Pinning only `status` has the mirror problem. noSyncTestReadsTheRealSyncState() is the guard, modelled on noTestCanSeeTheRealLockTable(). And Qt::ISODate carries no milliseconds. The status file is preferred only when it describes THIS run, compared against when the lock appeared, so a stale success cannot outrank a fresh failure; but the script writes date -Iseconds, and a round trip of "now" comes back 329 ms behind, measured. A fast sync's own file therefore parsed as stale and fell back to the log, with nothing failing to say so. One second of slack matches the precision the format carries. Design: docs/superpowers/specs/2026-08-29-sync-status-file-design.md Suite: 43 of 44, with undoMovesTheMessageBack failing as it does on master (item 136).
Diffstat (limited to 'docs/superpowers/specs/2026-08-29-sync-status-file-design.md')
-rw-r--r--docs/superpowers/specs/2026-08-29-sync-status-file-design.md177
1 files changed, 177 insertions, 0 deletions
diff --git a/docs/superpowers/specs/2026-08-29-sync-status-file-design.md b/docs/superpowers/specs/2026-08-29-sync-status-file-design.md
new file mode 100644
index 0000000..b5c3a40
--- /dev/null
+++ b/docs/superpowers/specs/2026-08-29-sync-status-file-design.md
@@ -0,0 +1,177 @@
+# A status file, so qtmaildir is told about a sync instead of inferring it
+
+Item 174, and item 125 with it. 2026-08-29.
+
+## The premise, which is the user's and which reverses the usual direction
+
+`assets/mailsync.sh` was written for another system and adapted for qtmaildir.
+It is now the only consumer, so the script serves the application rather than
+the application accommodating the script. Where a gap is found in qtmaildir,
+the script is reshaped to bridge it. mbsync and IMAP stay out of the
+application; the script stays a commodity.
+
+Two facts settle the scope, both verified rather than assumed:
+
+- The user's crontab runs `mailsync.sh` every ten minutes and nothing else
+ touches mail. There is no third-party mbsync and no bare `notmuch new`.
+- `~/bin/mailsync.sh` is a SYMLINK to `assets/mailsync.sh` in this repo, so an
+ edit here is live on the next cron tick with no deploy step.
+
+## What is actually wrong
+
+qtmaildir learns about a sync it did not start through two indirect channels,
+both of which are inferences about a process that has already exited:
+
+- **An inode in `/proc/locks`** (`syncmonitor.cpp:64`), watching the file the
+ script flocks. This answers "is a sync running", and it is the reason item
+ 125 exists: a skipped run (exit 75) releases no lock the monitor ever saw
+ held, so the spinner runs for ever.
+- **A grep of the log** for `RUN END ... status=OK`
+ (`MailSync::lastRunOutcome`, one production caller at `mainwindow.cpp:5182`).
+ This makes a human-readable log line into load-bearing wire format: anyone
+ reformatting that banner breaks the application silently.
+
+Neither channel carries what the application needs, which is WHICH CHANNELS a
+run synced, WHEN, and WITH WHAT RESULT. The consequence is visible in the code:
+the local sync path narrows its clear to the accounts the run carried
+(`mainwindow.cpp:4623`, `subtract(accountsThisRunCarried)`), while the external
+path cannot and does a blanket `m_pendingTagEdits.clear()` at `:5183`. The
+external path is coarser than the local one for want of information the script
+has and does not report.
+
+## What is NOT wrong, and must not be "fixed"
+
+**A bare `notmuch new` cannot clear the pending count, and should not.** The
+count means "confirmed tag mutations not yet known to have reached the MAIL
+STORE" (`mainwindow.h:1763`), which is the server. A tag edit is in notmuch the
+moment it is made; what is outstanding is mbsync pushing the renamed Maildir
+files. `notmuch new` re-indexes local files and pushes nothing.
+
+Item 174's own "Approach" section proposes watching
+`notmuch_database_get_revision()`. That is rejected here: a revision moves when
+mail ARRIVES as well as when our edits land, and in neither case does it say
+anything about the server. Clearing on a revision bump would make the indicator
+claim work is safe to quit on when it is still local, which is item 28's defect
+returning in a new costume. The item's own constraint gestures at this and then
+resolves it with a per-message re-check, but that check reads notmuch, which
+also cannot see the server.
+
+The status file is the correct instrument precisely because the script knows
+what notmuch cannot: whether MBSYNC ran and what it returned.
+
+## The design
+
+The script writes one JSON file at the end of every run, including a skipped
+one. qtmaildir watches that file.
+
+**Path:** `~/.local/state/qtmaildir/syncstatus.json`, configurable beside
+`syncLog` for the same reasons that key exists. Note it goes under the
+application's own state directory, not beside `mailsync.log` in
+`~/.local/state/`: the log is the script's, the status file is the interface.
+
+**Shape:**
+
+```json
+{
+ "version": 1,
+ "run_id": "2026-08-29T10:33:07+02:00",
+ "started": "2026-08-29T10:33:07+02:00",
+ "ended": "2026-08-29T10:33:41+02:00",
+ "state": "ok",
+ "channels": ["work", "personal"],
+ "mbsync_status": 0,
+ "notmuch_status": 0
+}
+```
+
+- `state` is `ok`, `failed` or `skipped`. Three states, not a boolean, because
+ a skip is neither: item 125 exists because a skip currently reads as neither
+ success nor failure and so resolves nothing.
+- `channels` is the channel list the run actually synced, or `["-a"]` for a
+ full run. This is what lets the external path narrow its clear the way the
+ local path already does.
+- `version` is refused rather than guessed at by a reader that does not know
+ it, following the rule the rules file already uses for `kFormatVersion`.
+
+**Written atomically**, to a temporary file in the same directory and then
+`mv`, which is atomic within a filesystem. A reader watching the file must
+never see a half-written one, and qtmaildir will be watching it while it is
+written.
+
+**The log keeps its banner.** `RUN END` stays exactly as it is, and
+`MailSync::lastRunOutcome()` stays with its eight tests. It becomes the
+FALLBACK for a status file that is missing or unreadable, which is what a first
+run after upgrading looks like, rather than being deleted in the same change
+that adds its replacement.
+
+## What each side does
+
+**`assets/mailsync.sh`.** Collect the channel list, write the file on every
+exit path. The skip branch at the top exits before the status directory exists,
+so it needs its own write. Two properties recorded as load-bearing survive
+untouched: printing to stdout as well as the log, and exiting with the real
+status.
+
+**`src/mailsync.{h,cpp}`.** A `SyncStatus` value struct and a static reader,
+beside `lastRunOutcome()` and in the same shape: a pure function of a path,
+returning a value with an Unknown-equivalent for every failure. No widget, no
+process.
+
+**`src/syncmonitor.{h,cpp}`.** Watch the status file with `QFileSystemWatcher`,
+beside the existing `/proc/locks` polling rather than replacing it. The lock
+answers "a sync is running now" and the file answers "a run finished and here
+is what it did"; these are different questions and a completion record cannot
+answer the first.
+
+**`src/mainwindow.cpp`.** The external path at `:5182` reads the status file
+instead of the log, and narrows its clear by channel, resolving channels back
+to accounts through `Account::channel`. A `skipped` state clears the spinner
+without clearing the count, which is item 125.
+
+## Constraints
+
+- **The script is live.** Every edit reaches real mail within ten minutes.
+ Writing the status file is purely additive and nothing reads it until the
+ application does, so a partial state is harmless, but the ORDER matters: the
+ script lands first and is left to run for a few cycles before the reader is
+ written.
+- **`Account::channel` may differ from the account key** and may be empty,
+ in which case the key is the channel. A reverse lookup must handle both, and
+ a channel naming no account must be ignored rather than dropped on the floor
+ silently.
+- **`-a` means every account**, and is not the same as a list naming them all.
+ A reader that treats `["-a"]` as an unknown channel clears nothing on exactly
+ the run that carried everything.
+- **Only a successful run may clear the count**, per the rule the local path
+ states at `:4614`. `failed` and `skipped` clear no edits.
+- **Unknown changes no state**, exactly as `SyncOutcome::Unknown` and
+ `SyncMonitor::State::Unknown` are treated today.
+- **Two readers, one format.** This is the `rules.json` situation again, a bash
+ writer and a C++ reader agreeing by test rather than by shared code. The
+ discipline in AGENTS.md under "Changing the rule format" applies: change both
+ sides together, bump the version only for a breaking change, and run both
+ suites.
+- **No new dependency for JSON.** The script writes it with `printf`, which is
+ why the shape above has no nesting; qtmaildir reads it with `QJsonDocument`,
+ which it already uses for `queries.json`.
+
+## Tests
+
+- `assets/hooks/`-style Python test for the script, in the shape the two hook
+ suites already take: run it with a stub `mbsync` and `notmuch` on PATH,
+ assert the file's contents for ok, failed and skipped.
+- `test_mailsync.cpp`: the reader, against files written by hand. Every failure
+ mode returns the Unknown equivalent, including a truncated file, an unknown
+ version, and a file that is not JSON at all.
+- `test_mainwindow.cpp`: the external path clears the count for a channel it
+ carried and leaves an account it did not, which is the behaviour the blanket
+ clear cannot express. And a `skipped` run clears the spinner and no edits.
+
+## What this deliberately does not do
+
+- No daemon, no socket, no D-Bus. The file is the interface.
+- No change to how a LOCAL sync reports: it has the process and its exit
+ status, which is better evidence than a file.
+- No backfill of item 125's other half. The spinner is cleared by a `skipped`
+ record here; whether `SyncMonitor` should also time out an observation it
+ never saw end is a separate question and stays in item 125.