Initial commit: umbrella design for abusectl

An abuse reporting sidecar for phishing mail: parse a flagged message,
extract its indicators, resolve abuse contacts, and file the result to a
MISP instance and to public abuse channels.

This is the umbrella spec, agreed in one design session. Each part gets
its own spec before it is built; this settles what the parts share and
what would be expensive to change later: the case directory and its
manifest format, the redaction rule, the ordering between MISP and the
vendors, and how partial failure is recorded.

It exists as a separate tool because qtmaildir does no network protocol
work by design, and this needs RDAP, three vendor APIs and mail to abuse
desks. qtmaildir invokes it by name the way it invokes mailsync.sh, and
hosts the review dialog; the two are coupled only by the manifest format
and a command name in config.

Two properties are recorded as safety properties rather than
preferences. Recipient identifiers are never captured, at extraction
rather than at submission, so the tool cannot disclose an identifier it
was never given; tracking tokens inside URLs are covered, since a
parameter value is frequently the recipient's address. And nothing
remote is fetched while parsing, because following a link confirms the
address is live and fires the tracker.

parse is the first part to build: stdlib only, no network, no config,
and its output is the format every other part reads.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R

1 files changed, 423 insertions, 0 deletions

diff --git a/docs/specs/2026-09-08-abusectl-design.md b/docs/specs/2026-09-08-abusectl-design.md
new file mode 100644
index 0000000..7f72b5c
--- /dev/null
+++ b/docs/specs/2026-09-08-abusectl-design.md
@@ -0,0 +1,423 @@
+# abusectl: abuse reporting sidecar, umbrella design
+
+Status: **agreed 2026-09-08**, in one brainstorming session with the user.
+This is the UMBRELLA spec. Each part gets its own spec before it is built;
+this one settles the decisions the parts share and would be expensive to
+change later.
+
+Backlog: item 194 in qtmaildir's
+`docs/superpowers/plans/2026-08-03-post-0.1.0-usability.md`, which carries the
+qtmaildir half and its ordering against items 187 and 190. This document is
+the sidecar, and the two repositories are coupled only by the manifest format
+below and the command name in qtmaildir's config.
+
+## Why this exists
+
+The user is a cybersecurity consultant. They have been phished, and they want
+to act quickly and effectively on a campaign that targets them: parse a
+flagged message, extract its indicators, find who to report each one to, and
+file the result both to their own MISP instance and to the public abuse
+channels. Filling a database with phishing attempts is a feature no other mail
+client offers, which is a reason this one exists.
+
+Two of the user's own constraints are SAFETY PROPERTIES rather than
+preferences, stated when the item was first written and not to be traded away
+for convenience. They are specified in full below: recipient identifiers are
+never captured, and no remote content is ever fetched during parsing.
+
+## Why it is not in qtmaildir
+
+qtmaildir does no network protocol work at all, by design: fetching and
+sending are external scripts, which is why there is no IMAP client, no SMTP
+client, and why send is a per-account `send_command` on stdin. This tool needs
+RDAP, three vendor REST APIs and mail to abuse desks, which is four outbound
+protocols. Building it into `src/` would repeal that rule rather than extend
+it. Its ecosystem is Python besides, where `assets/hooks/` already puts the
+non-GUI parts of this mail system.
+
+**The split is architectural and is not a judgement on the feature.** The
+review UI lives in qtmaildir, because a reviewable document is exactly what
+makes a GUI review possible.
+
+**A separate repository, not a submodule.** qtmaildir invokes `abusectl` as a
+subprocess by name, the way it already invokes `mailsync.sh` and the
+`send_command`; nothing in `src/` includes or imports it. A submodule pins a
+commit that is not the one that runs, since what executes is whatever is
+installed on `$PATH`, and it would cost a two-step clone, an ordering
+constraint on every push across two remotes, and an empty directory in the
+release tarball, because `git archive` does not include submodule contents.
+Compatibility, if it ever matters, is a `format` version in the manifest, the
+same discipline `rules.json` already uses across two readers in two languages.
+
+## Shape
+
+One tool, subcommands, one repository. Each subcommand reads and writes a
+**case directory**, which is where state lives between steps. A review can
+therefore take a week and survive a reboot.
+
+```
+abusectl parse msg.eml        -> case dir, IOCs        offline, pure
+abusectl contacts <case>      -> + abuse contacts      network, read-only
+abusectl report <case>        -> + report bodies       offline, pure
+        [ review, in qtmaildir or $EDITOR ]
+abusectl submit <case>        -> MISP, then vendors    network, writes
+```
+
+Every subcommand is independently runnable and independently useful. `parse`
+alone triages a message. `parse` + `contacts` + `report` produces a document
+that can be sent by hand, with no API key configured anywhere.
+
+**The tool stays usable from a terminal.** The qtmaildir dialog is one front
+end, never the only one: a message that never came through qtmaildir, or a
+session over SSH, must still be workable. Same reasoning as `mailsync.sh`
+printing to stdout as well as its log.
+
+## The one hard ordering rule
+
+`submit` writes **MISP first** and aborts the vendor fan-out if that write
+fails.
+
+MISP is the user's own instance and is correctable through its REST API
+(update, soft-delete, hard-delete); a submission to AbuseIPDB, URLhaus or
+VirusTotal cannot be recalled. So the reversible step gates the irreversible
+ones. Reporting externally with no local record of having done so, and no
+dedup entry to stop a duplicate later, is worse than not reporting at all.
+
+MISP is also the DEDUP ORACLE. It correlates attributes by value across
+events and answers `/attributes/restSearch`, so `submit` can ask whether the
+infrastructure has been reported before and skip or attach rather than
+duplicate. Writing it before the fan-out is what makes that answer available.
+
+**Correction is MISP's own web UI**, not a subcommand here. A destructive verb
+in a tool run often, guarding data the user cares about, is not worth building
+for something MISP already does well. A correction subcommand is possible
+later.
+
+## The case directory
+
+```
+<cases>/2026-09-08-a3f1/
+  source.eml          the original, unredacted
+  manifest.json       IOCs, contacts, destinations, per-destination status
+  bodies/
+    abusedb.json
+    rdap-1.xarf
+```
+
+The path is configurable. **Nothing deletes cases**: they are the user's
+evidence, and a tool that silently bins a report sent last month is worse than
+a directory that grows. Cleanup is not built.
+
+`manifest.json` carries a `format` version. Bodies are separate files because
+they are text a human edits; putting them in the manifest would mean editing
+escaped strings inside a JSON document, or growing an extract/apply pair,
+which is a directory reinvented with extra steps.
+
+**`source.eml` is unredacted local evidence, and the submit path must never
+attach it wholesale.** This is the one route by which the redaction guarantee
+below could leak, so it is stated rather than left obvious. The case directory
+is sensitive at rest.
+
+## Redaction, structurally
+
+**The parser never captures recipient identifiers.** Not stored, not hashed,
+not written to the manifest:
+
+- `To`, `Cc`, `Delivered-To`, `X-Original-To`
+- the user's own Message-IDs
+- maildir paths and account keys
+
+The guarantee is that **the tool cannot disclose an identifier it was never
+given**. It holds no matter what a later subcommand does, which is why it is
+enforced at extraction rather than at submission: a strip-on-submit rule
+depends on every future sending path remembering, and this one does not.
+
+**The report is reviewable and hand-editable before anything is sent.** What
+the tool derives and what the user chooses to disclose are separate things.
+The user can add facts during review that the tool never held, including how
+many of their addresses a campaign hit, which is a fact they supply from their
+own knowledge.
+
+### Tracking parameters are recipient identifiers
+
+A phishing URL commonly carries the recipient's identity in its query string:
+`?e=you@example.org` plaintext, `?u=<base64 of the address>`, or
+`?id=<md5 of the address>`, which a wordlist or a targeted guess reverses.
+Publishing that to four third parties is the same leak as publishing the `To`
+header, hidden one level down.
+
+It also **deanonymises the reporter to the attacker**. Abuse desks forward
+reports to their customers and URLhaus is a public feed, so a kit operator
+watching for their own URLs learns which target reported them. For a
+consultant the parameter may carry a CLIENT's identifier rather than the
+user's own.
+
+**So: keep scheme, host and path; keep parameter NAMES; redact parameter
+VALUES.**
+
+```
+http://login-example.invalid/verify?id=REDACTED&src=REDACTED
+```
+
+This costs the report almost nothing. Parameter names are part of the kit's
+fingerprint and campaigns correlate on infrastructure, not on per-victim
+tokens. The token is unique per recipient BY DESIGN, so it is the one part of
+the URL that cannot correlate anything, and keeping it makes dedup actively
+worse: two messages from one campaign would look like different URLs.
+
+**Path segments carry the token too.** `/verify/ZGFuaXhAZXhhbXBsZS5vcmc/` is
+common and a query-string-only rule misses it entirely. A path segment that
+looks like base64 or a long hex string is FLAGGED FOR REVIEW rather than
+silently redacted, since a path may also be meaningful.
+
+The full URL survives in `source.eml` either way, so the evidence exists
+locally; it is simply not what gets published by default, and the review gate
+allows pasting one in by hand when a particular desk genuinely needs it.
+
+## Components
+
+```
+abusectl/
+  cli.py          argparse dispatch, exit codes. No logic.
+  case.py         case dir: create, load, save manifest, atomic writes
+  parse.py        .eml -> IOCs                       stdlib only, pure
+  contacts.py     IOCs -> abuse contacts (RDAP)      network, read-only
+  report.py       IOCs + contacts -> bodies          pure
+  submit.py       bodies -> MISP, then vendors       network, writes
+  destinations/   one module per target
+    misp.py  abusedb.py  urlhaus.py  virustotal.py  email.py
+  config.py       ~/.config/abusectl/config.toml
+```
+
+**`case.py` is the spine.** Every subcommand goes through it and nothing else
+writes the case directory. It owns the manifest schema and its version, and it
+writes ATOMICALLY, temp file plus rename, the way `mailrules.py` saves
+`rules.json`. A half-written manifest during a review is a corrupted evidence
+record.
+
+**`parse.py` is stdlib-only and pure.** Bytes in, IOC list out; no socket, no
+config read. That is what makes it testable against fixtures with no setup.
+
+**`contacts.py` and `submit.py` are the only network modules**, and both take
+an INJECTED TRANSPORT. Not an abstraction for its own sake: it is the seam
+that lets the irreversible path be tested without sending anything.
+
+**`destinations/` is one module per target**, each exposing the same two
+functions, build a body and send one. A fifth vendor is a new file rather than
+an edit to `submit.py`. This is the one place a plugin shape earns itself,
+because there are four known members with genuinely different APIs. There is
+no discovery mechanism; it is a package with four members.
+
+**`config.py`** reads TOML through stdlib `tomllib`, no dependency. It holds
+the cases path, the MISP URL and key, vendor keys, the user's reporting
+identity for X-ARF, and the trusted-relay boundary described below. Secrets
+live in a file with mode `0600`, checked on load, and **never reach the
+manifest or a log**.
+
+### Deliberately absent
+
+No database of its own: MISP is the database and case directories are the
+local record. No daemon, no queue, no retry scheduler, since retry is the user
+running `submit` again.
+
+### Dependencies by part
+
+| Part | Needs |
+|---|---|
+| `parse` | stdlib only |
+| `contacts` | an HTTP client |
+| `report` | stdlib only |
+| `submit` | HTTP client, PyMISP |
+
+`requirements.txt` is empty until `contacts` is built. **Development runs from
+a venv in the repository**, gitignored; packaging and distribution are
+deliberately out of scope until the tool does something worth installing. A
+SlackBuild in `my-slackbuilds` is the eventual shape, with an nvchecker
+stanza, matching every other tool the user runs.
+
+## Data flow
+
+```
+msg.eml
+   |  parse      creates case, writes source.eml + iocs[]
+   v
+case dir --------------------------------------------------+
+   |  contacts   RDAP per IOC, writes contacts[]            |
+   v                                                        |  every step
+case dir                                                    |  reads and
+   |  report     writes bodies/, destinations[]             |  rewrites
+   v                                                        |  manifest.json
+case dir  -- [ REVIEW: qtmaildir dialog or $EDITOR ] -------+
+   |  submit                                                |
+   +--> MISP ------ fails? STOP, nothing sent --------------+
+   +--> vendors + abuse desks, per-destination status
+```
+
+**Review edits the BODIES, not the IOCs.** Once `report` has run the bodies
+are the artifact, so `report` refuses to run again on a case whose bodies were
+modified after generation unless forced, by a timestamp check in the manifest.
+Silently discarding a review that took twenty minutes is what makes a tool
+untrustworthy.
+
+**`submit` is resumable.** It reads per-destination status and skips anything
+already `sent`, so running it twice is safe. This matters because the failure
+that will actually be hit is a rate limit, not a bug.
+
+## Manifest schema
+
+```json
+{
+  "format": 1,
+  "case_id": "2026-09-08-a3f1",
+  "created": "2026-09-08T12:31:04Z",
+  "source": "source.eml",
+
+  "iocs": [
+    { "id": "ioc-1", "type": "ipv4", "value": "203.0.113.42",
+      "origin": "received-chain", "confidence": "untrusted-hop" },
+    { "id": "ioc-2", "type": "domain", "value": "login-example.invalid",
+      "origin": "href" },
+    { "id": "ioc-3", "type": "url",
+      "value": "http://login-example.invalid/verify?id=REDACTED",
+      "origin": "href-html" },
+    { "id": "ioc-4", "type": "sha256", "value": "e3b0c442...",
+      "origin": "attachment", "filename": "invoice.pdf" }
+  ],
+
+  "auth": { "spf": "fail", "dkim": "none", "dmarc": "fail" },
+
+  "contacts": [
+    { "ioc": "ioc-1", "abuse": "abuse@example.invalid",
+      "source": "rdap", "handle": "AS64496" },
+    { "ioc": "ioc-2", "abuse": null, "source": "rdap",
+      "error": "no abuse contact published" }
+  ],
+
+  "destinations": [
+    { "id": "misp",    "kind": "misp",  "iocs": ["ioc-1","ioc-2","ioc-3"],
+      "body": null,                    "status": "pending" },
+    { "id": "abusedb", "kind": "api",   "iocs": ["ioc-1"],
+      "body": "bodies/abusedb.json",   "status": "pending" },
+    { "id": "rdap-1",  "kind": "email", "iocs": ["ioc-1"],
+      "target": "abuse@example.invalid",
+      "body": "bodies/rdap-1.xarf",    "status": "pending" }
+  ]
+}
+```
+
+After a submit in which one destination failed:
+
+```json
+{ "id": "abusedb", "status": "sent",
+  "sent_at": "2026-09-08T12:40:11Z", "receipt": "8891234" },
+{ "id": "virustotal", "status": "failed",
+  "attempted_at": "2026-09-08T12:40:12Z", "error": "429 rate limited" }
+```
+
+### Five schema decisions, and why
+
+**IOCs carry an `id` and everything references it.** Contacts and destinations
+point at `ioc-2` rather than repeating the value, so there is one place to
+correct it and a destination cannot drift from the IOC it reports.
+
+**`origin` on every IOC** names where it came from: `received-chain`, `href`,
+`attachment`. During review the user needs to know whether an IP came from a
+header to trust or one the attacker wrote. Without it, review is guesswork.
+
+**`confidence: untrusted-hop`** exists because of the `Received` trap below.
+An IP from beneath the trusted-relay boundary is attacker-supplied, and the
+manifest says so rather than presenting it as fact.
+
+**`status` is per destination**, never one status for the case. There is no
+single answer when four destinations disagree.
+
+**MISP is a destination like the others**, with `body: null` because PyMISP
+builds its own payload. That keeps `submit` one loop with one ordering rule
+rather than a special case beside a loop.
+
+## Error handling
+
+**Partial failure is the normal case, not an exception.** Four destinations
+can disagree, so each carries its own status and a retry sends only what did
+not land. All-or-nothing would be wrong here: the successful submissions
+really happened and cannot be recalled, so recording them as failed would make
+the next attempt duplicate real reports.
+
+**A MISP failure aborts before anything irreversible runs**, per the ordering
+rule above.
+
+**A missing abuse contact is not an error.** RDAP publishes none for many
+netblocks. The IOC keeps `"abuse": null` with the reason, the destination is
+simply not created, and review shows what could not be resolved.
+
+## Testing
+
+`parse` is fixtures in, JSON out, so the whole reversible half is testable
+offline with no network and no keys. The network parts get their transport
+stubbed through the injected seam.
+
+**Fixtures use `example.org` and `.invalid` addresses and generic
+placeholders**, per the user's standing personal-data rule. Real phishing
+samples do not go in the repository: they carry the recipient identifiers this
+tool exists to keep out of reports, and a repository is potentially public.
+
+Python conventions follow `assets/hooks/` in qtmaildir: GPLv2 header on every
+file, `unittest`, a `test_<module>.py` beside each module, no framework.
+
+## `parse` in detail: the first part to build
+
+Extracted:
+
+- **Sending IP** from the `Received` chain: the last UNTRUSTED hop
+- **Sender domains**: `Return-Path`, `From`, `Reply-To` where it differs
+- **Href domains and full URLs** from HTML and text parts, decoded, redacted
+  per the rule above, never fetched
+- **Redirect chains** as DECLARED in headers and href text, never followed
+- **Auth results**: `Authentication-Results`, and the SPF/DKIM/DMARC verdicts
+  as the receiving server recorded them
+- **Attachment filenames and SHA-256 hashes**, since a hash is reportable and
+  a filename is an indicator
+
+Never captured: recipients, the user's Message-IDs, maildir paths, account
+keys.
+
+### Two traps, which is where this kind of parser goes wrong
+
+**The `Received` chain is attacker-controlled below the user's own
+infrastructure.** Headers can be forged wholesale, and only the hops the
+user's own MTA added are trustworthy. Without a configured trusted-relay
+boundary, the "sending IP" is whatever the attacker chose to write. The
+boundary is config; IOCs below it are marked `untrusted-hop` rather than
+presented as fact.
+
+**Never resolve and never fetch.** Not the URLs, not the redirects, not remote
+images. Following a link confirms the address is live to the sender and fires
+exactly the tracker the message wanted. This is a parser over bytes, and that
+is a safety property rather than a performance choice.
+
+## The qtmaildir half
+
+Specified in item 194 and built after 187 and 190, which settle what Mark spam
+does and put it on the message bar. It is a review dialog over the case
+directory: show the bodies, allow editing, show what could not be resolved,
+and call `submit` on approval, then show per-destination results with a retry
+for the failures. Sized M in the backlog, not the S a plain button would be:
+a review dialog is real UI, and the user asked for the review to happen in the
+application rather than in an editor.
+
+The wire contract is this document's manifest plus the `abusectl` command name
+in qtmaildir's config, in the shape `[sync] command` and the per-account
+`send_command` already use.
+
+## What each later spec has to settle
+
+- **`contacts`**: RDAP bootstrap and referral chasing, caching policy, rate
+  limits, and what happens for a netblock that publishes no abuse contact.
+- **`report`**: the X-ARF (RFC 5965) schema version, which fields the user's
+  reporting identity fills, and the plain-text alternative for desks that do
+  not parse X-ARF.
+- **`submit`**: per-vendor auth and payload shapes, MISP event structure
+  (one event per case, or per campaign, and how the dedup query decides),
+  and how mail to an abuse desk is sent, which most likely reuses qtmaildir's
+  own `send_command` rather than adding an SMTP client.
+- **The qtmaildir dialog**: its own item, after 187 and 190.