spec: retry deferred destinations from cron, not by hand

A rate limit tells us when, not merely that: a 429 carries Retry-After or
the vendor's reset headers. Recording the deadline and asking the user to
run submit again throws that away and relies on them remembering.

So `abusectl retry` scans every case for destinations whose retry_after
has passed and sends only those, as one unattended cron line beside
mailsync.sh. No inline retry: submit never sleeps waiting for a window,
because a daily quota resets in hours and a process killed while
sleeping is back to the user remembering. One mechanism, not two.

Unattended retry makes three properties load-bearing, since a retry that
re-sends is a duplicate abuse report and cannot be withdrawn. Status is
written before the attempt, so a crash mid-send leaves in-flight, which
is honest, rather than looking like it never happened; retry never
touches in-flight. Attempts are capped, so a dead abuse mailbox stops
being retried. A soft failure with no server deadline gets exponential
backoff.

deferred and failed are separate statuses: deferred means the tool will
handle it, failed means the user must. Collapsing them either strands a
rate-limited report forever or retries a dead mailbox indefinitely.

The qtmaildir dialog accordingly grows no retry button. A deferred
destination belongs to cron, and a button beside it would race the
scheduled run.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R

1 files changed, 123 insertions, 12 deletions

diff --git a/docs/specs/2026-09-08-abusectl-design.md b/docs/specs/2026-09-08-abusectl-design.md
index 7f72b5c..ed984dd 100644
--- a/docs/specs/2026-09-08-abusectl-design.md
+++ b/docs/specs/2026-09-08-abusectl-design.md
@@ -61,6 +61,7 @@ abusectl contacts <case>      -> + abuse contacts      network, read-only
 abusectl report <case>        -> + report bodies       offline, pure
         [ review, in qtmaildir or $EDITOR ]
 abusectl submit <case>        -> MISP, then vendors    network, writes
+abusectl retry                -> whatever is due       network, writes, cron
 ```
 
 Every subcommand is independently runnable and independently useful. `parse`
@@ -184,6 +185,7 @@ abusectl/
   contacts.py     IOCs -> abuse contacts (RDAP)      network, read-only
   report.py       IOCs + contacts -> bodies          pure
   submit.py       bodies -> MISP, then vendors       network, writes
+  retry.py        due destinations, across cases     network, writes
   destinations/   one module per target
     misp.py  abusedb.py  urlhaus.py  virustotal.py  email.py
   config.py       ~/.config/abusectl/config.toml
@@ -198,8 +200,8 @@ record.
 **`parse.py` is stdlib-only and pure.** Bytes in, IOC list out; no socket, no
 config read. That is what makes it testable against fixtures with no setup.
 
-**`contacts.py` and `submit.py` are the only network modules**, and both take
-an INJECTED TRANSPORT. Not an abstraction for its own sake: it is the seam
+**`contacts.py`, `submit.py` and `retry.py` are the only network modules**,
+and each takes an INJECTED TRANSPORT. Not an abstraction for its own sake: it is the seam
 that lets the irreversible path be tested without sending anything.
 
 **`destinations/` is one module per target**, each exposing the same two
@@ -210,15 +212,17 @@ no discovery mechanism; it is a package with four members.
 
 **`config.py`** reads TOML through stdlib `tomllib`, no dependency. It holds
 the cases path, the MISP URL and key, vendor keys, the user's reporting
-identity for X-ARF, and the trusted-relay boundary described below. Secrets
+identity for X-ARF, `max_attempts` for the retry cap, and the trusted-relay
+boundary described below. Secrets
 live in a file with mode `0600`, checked on load, and **never reach the
 manifest or a log**.
 
 ### Deliberately absent
 
 No database of its own: MISP is the database and case directories are the
-local record. No daemon, no queue, no retry scheduler, since retry is the user
-running `submit` again.
+local record. **No daemon and no queue.** Retry is a `retry` subcommand run
+from cron, described below; the scheduling belongs to cron, which the user
+already runs `mailsync.sh` from, and not to a process of this tool's own.
 
 ### Dependencies by part
 
@@ -228,6 +232,7 @@ running `submit` again.
 | `contacts` | an HTTP client |
 | `report` | stdlib only |
 | `submit` | HTTP client, PyMISP |
+| `retry` | HTTP client (no PyMISP: it never writes MISP) |
 
 `requirements.txt` is empty until `contacts` is built. **Development runs from
 a venv in the repository**, gitignored; packaging and distribution are
@@ -251,6 +256,12 @@ case dir  -- [ REVIEW: qtmaildir dialog or $EDITOR ] -------+
    |  submit                                                |
    +--> MISP ------ fails? STOP, nothing sent --------------+
    +--> vendors + abuse desks, per-destination status
+             |
+             +--> rate limited? status `deferred` + retry_after
+                        |
+                        v
+                  cron: abusectl retry     every N minutes,
+                  sends only what is due   across every case
 ```
 
 **Review edits the BODIES, not the IOCs.** Once `report` has run the bodies
@@ -261,7 +272,8 @@ untrustworthy.
 
 **`submit` is resumable.** It reads per-destination status and skips anything
 already `sent`, so running it twice is safe. This matters because the failure
-that will actually be hit is a rate limit, not a bug.
+that will actually be hit is a rate limit, not a bug, and because `retry`
+below runs unattended over the same cases.
 
 ## Manifest schema
 
@@ -305,15 +317,38 @@ that will actually be hit is a rate limit, not a bug.
 }
 ```
 
-After a submit in which one destination failed:
+After a submit in which one destination landed, one was rate limited and one
+is genuinely broken:
 
 ```json
 { "id": "abusedb", "status": "sent",
   "sent_at": "2026-09-08T12:40:11Z", "receipt": "8891234" },
-{ "id": "virustotal", "status": "failed",
-  "attempted_at": "2026-09-08T12:40:12Z", "error": "429 rate limited" }
+
+{ "id": "virustotal", "status": "deferred",
+  "attempted_at": "2026-09-08T12:40:12Z",
+  "retry_after": "2026-09-08T13:00:00Z",
+  "attempts": 1, "error": "429 rate limited" },
+
+{ "id": "rdap-1", "status": "failed",
+  "attempted_at": "2026-09-08T12:40:13Z",
+  "attempts": 5, "error": "550 no such mailbox" }
 ```
 
+### The five statuses
+
+| status | meaning | who acts |
+|---|---|---|
+| `pending` | never attempted | `submit` |
+| `in-flight` | attempt started, outcome unknown | a human |
+| `sent` | landed, receipt recorded | nobody, ever again |
+| `deferred` | soft failure, `retry_after` set | `retry`, from cron |
+| `failed` | hard failure, or attempts exhausted | a human |
+
+`deferred` and `failed` are deliberately distinct. Deferred means the tool
+will handle it; failed means it will not, and the user has to look. Collapsing
+them either strands a rate-limited report forever or retries a dead mailbox
+until the end of time.
+
 ### Five schema decisions, and why
 
 **IOCs carry an `id` and everything references it.** Contacts and destinations
@@ -350,6 +385,75 @@ rule above.
 netblocks. The IOC keeps `"abuse": null` with the reason, the destination is
 simply not created, and review shows what could not be resolved.
 
+## Retry, from cron
+
+A rate limit tells us WHEN, not merely that: a 429 carries `Retry-After`, or
+the vendor's rate-limit reset headers do. Discarding that and asking the user
+to run `submit` again throws away a fact the server handed us, and relies on
+them remembering. So the deadline is recorded and cron acts on it.
+
+```
+*/N * * * *  abusectl retry
+```
+
+`abusectl retry` scans every case under the configured cases directory, finds
+destinations whose status is `deferred` and whose `retry_after` has passed,
+and sends only those. It takes no case argument: the point is that it needs no
+knowledge of what is outstanding, so it can be a single unattended cron line
+beside `mailsync.sh`.
+
+**There is no inline retry.** `submit` never sleeps waiting for a window to
+open. A foreground process blocked for an unknown period is worse than the
+problem it solves, since a daily quota resets in hours rather than seconds,
+and a process killed while sleeping is back to the user remembering. One
+mechanism, not two.
+
+### What makes unattended retry safe
+
+Automated retry turns three properties from conveniences into load-bearing
+ones. A retry that re-sends is a duplicate abuse report, which cannot be
+withdrawn.
+
+**Status is written BEFORE the attempt, not only after.** A destination is
+marked `in-flight` with its manifest flushed to disk, then attempted, then
+marked `sent` or `deferred`. A crash between the send and the status write
+therefore leaves `in-flight`, which is honest: the report may or may not have
+arrived. **`retry` never touches an `in-flight` destination**; it is for a
+human to resolve by checking the receiving end. The alternative, writing only
+after, makes an interrupted send indistinguishable from one that never
+happened, and cron would re-send it.
+
+**`attempts` is counted and capped.** A destination that has failed
+`max_attempts` times (config, default 5) becomes `failed` rather than
+`deferred`, so a permanently broken abuse mailbox is not retried forever. The
+count is per destination, not per case.
+
+**Backoff on a deadline the server did not give.** Where a failure is soft but
+carries no `Retry-After`, `retry_after` is set by exponential backoff from
+`attempts`, so a flapping endpoint is not hammered every N minutes.
+
+### What retry must never do
+
+- **Never re-send a `sent` destination.** The receipt is recorded; there is
+  nothing to redo.
+- **Never touch `in-flight`.** Ambiguous is not the same as failed.
+- **Never run the MISP ordering rule again.** MISP was written before the
+  fan-out during the original `submit`, so by the time a destination is
+  `deferred` the local record already exists. `retry` sends vendor and email
+  destinations only; a MISP write that failed aborted the submission before
+  any of them was attempted, so there is nothing deferred to pick up.
+- **Never re-read or re-generate the bodies.** It sends what was reviewed. A
+  body edited after submission is not silently picked up, for the same reason
+  `report` refuses to overwrite a reviewed case.
+
+### Concurrency
+
+`retry` runs from cron and the user may run `submit` by hand at the same
+moment. Each case directory is locked for the duration of an attempt, with a
+lock file inside it, and a case that is locked is skipped rather than waited
+for: the next cron tick will reach it. This is the same reasoning as
+`mailsync.sh` exiting 75 when another run holds its lock.
+
 ## Testing
 
 `parse` is fixtures in, JSON out, so the whole reversible half is testable
@@ -400,8 +504,13 @@ is a safety property rather than a performance choice.
 Specified in item 194 and built after 187 and 190, which settle what Mark spam
 does and put it on the message bar. It is a review dialog over the case
 directory: show the bodies, allow editing, show what could not be resolved,
-and call `submit` on approval, then show per-destination results with a retry
-for the failures. Sized M in the backlog, not the S a plain button would be:
+and call `submit` on approval, then show per-destination results.
+
+It offers no retry button. A `deferred` destination belongs to cron, and a
+button beside it would race the scheduled run and risk a duplicate report; the
+dialog says when the retry is due instead. A `failed` or `in-flight`
+destination needs the user to act somewhere else, not to press a button that
+would repeat what already did not work. Sized M in the backlog, not the S a plain button would be:
 a review dialog is real UI, and the user asked for the review to happen in the
 application rather than in an editor.
 
@@ -416,7 +525,9 @@ in qtmaildir's config, in the shape `[sync] command` and the per-account
 - **`report`**: the X-ARF (RFC 5965) schema version, which fields the user's
   reporting identity fills, and the plain-text alternative for desks that do
   not parse X-ARF.
-- **`submit`**: per-vendor auth and payload shapes, MISP event structure
+- **`submit` and `retry`**: per-vendor auth and payload shapes, which
+  rate-limit headers each vendor actually sends and how a soft failure is
+  told from a hard one, the backoff curve, MISP event structure
   (one event per case, or per campaign, and how the dedup query decides),
   and how mail to an abuse desk is sent, which most likely reuses qtmaildir's
   own `send_command` rather than adding an SMTP client.