# abusectl: abuse reporting sidecar, umbrella design Status: **agreed 2026-09-08**, in one brainstorming session with the user. This is the UMBRELLA spec. Each part gets its own spec before it is built; this one settles the decisions the parts share and would be expensive to change later. Backlog: item 194 in qtmaildir's `docs/superpowers/plans/2026-08-03-post-0.1.0-usability.md`, which carries the qtmaildir half and its ordering against items 187 and 190. This document is the sidecar, and the two repositories are coupled only by the manifest format below and the command name in qtmaildir's config. ## Why this exists The user is a cybersecurity consultant. They have been phished, and they want to act quickly and effectively on a campaign that targets them: parse a flagged message, extract its indicators, find who to report each one to, and file the result both to their own MISP instance and to the public abuse channels. Filling a database with phishing attempts is a feature no other mail client offers, which is a reason this one exists. Two of the user's own constraints are SAFETY PROPERTIES rather than preferences, stated when the item was first written and not to be traded away for convenience. They are specified in full below: recipient identifiers are never captured, and no remote content is ever fetched during parsing. ## Why it is not in qtmaildir qtmaildir does no network protocol work at all, by design: fetching and sending are external scripts, which is why there is no IMAP client, no SMTP client, and why send is a per-account `send_command` on stdin. This tool needs RDAP, three vendor REST APIs and mail to abuse desks, which is four outbound protocols. Building it into `src/` would repeal that rule rather than extend it. Its ecosystem is Python besides, where `assets/hooks/` already puts the non-GUI parts of this mail system. **The split is architectural and is not a judgement on the feature.** The review UI lives in qtmaildir, because a reviewable document is exactly what makes a GUI review possible. **A separate repository, not a submodule.** qtmaildir invokes `abusectl` as a subprocess by name, the way it already invokes `mailsync.sh` and the `send_command`; nothing in `src/` includes or imports it. A submodule pins a commit that is not the one that runs, since what executes is whatever is installed on `$PATH`, and it would cost a two-step clone, an ordering constraint on every push across two remotes, and an empty directory in the release tarball, because `git archive` does not include submodule contents. Compatibility, if it ever matters, is a `format` version in the manifest, the same discipline `rules.json` already uses across two readers in two languages. ## Shape One tool, subcommands, one repository. Each subcommand reads and writes a **case directory**, which is where state lives between steps. A review can therefore take a week and survive a reboot. ``` abusectl parse msg.eml -> case dir, IOCs offline, pure abusectl contacts -> + abuse contacts network, read-only abusectl report -> + report bodies offline, pure [ review, in qtmaildir or $EDITOR ] abusectl submit -> MISP, then vendors network, writes ``` Every subcommand is independently runnable and independently useful. `parse` alone triages a message. `parse` + `contacts` + `report` produces a document that can be sent by hand, with no API key configured anywhere. **The tool stays usable from a terminal.** The qtmaildir dialog is one front end, never the only one: a message that never came through qtmaildir, or a session over SSH, must still be workable. Same reasoning as `mailsync.sh` printing to stdout as well as its log. ## The one hard ordering rule `submit` writes **MISP first** and aborts the vendor fan-out if that write fails. MISP is the user's own instance and is correctable through its REST API (update, soft-delete, hard-delete); a submission to AbuseIPDB, URLhaus or VirusTotal cannot be recalled. So the reversible step gates the irreversible ones. Reporting externally with no local record of having done so, and no dedup entry to stop a duplicate later, is worse than not reporting at all. MISP is also the DEDUP ORACLE. It correlates attributes by value across events and answers `/attributes/restSearch`, so `submit` can ask whether the infrastructure has been reported before and skip or attach rather than duplicate. Writing it before the fan-out is what makes that answer available. **Correction is MISP's own web UI**, not a subcommand here. A destructive verb in a tool run often, guarding data the user cares about, is not worth building for something MISP already does well. A correction subcommand is possible later. ## The case directory ``` /2026-09-08-a3f1/ source.eml the original, unredacted manifest.json IOCs, contacts, destinations, per-destination status bodies/ abusedb.json rdap-1.xarf ``` The path is configurable. **Nothing deletes cases**: they are the user's evidence, and a tool that silently bins a report sent last month is worse than a directory that grows. Cleanup is not built. `manifest.json` carries a `format` version. Bodies are separate files because they are text a human edits; putting them in the manifest would mean editing escaped strings inside a JSON document, or growing an extract/apply pair, which is a directory reinvented with extra steps. **`source.eml` is unredacted local evidence, and the submit path must never attach it wholesale.** This is the one route by which the redaction guarantee below could leak, so it is stated rather than left obvious. The case directory is sensitive at rest. ## Redaction, structurally **The parser never captures recipient identifiers.** Not stored, not hashed, not written to the manifest: - `To`, `Cc`, `Delivered-To`, `X-Original-To` - the user's own Message-IDs - maildir paths and account keys The guarantee is that **the tool cannot disclose an identifier it was never given**. It holds no matter what a later subcommand does, which is why it is enforced at extraction rather than at submission: a strip-on-submit rule depends on every future sending path remembering, and this one does not. **The report is reviewable and hand-editable before anything is sent.** What the tool derives and what the user chooses to disclose are separate things. The user can add facts during review that the tool never held, including how many of their addresses a campaign hit, which is a fact they supply from their own knowledge. ### Tracking parameters are recipient identifiers A phishing URL commonly carries the recipient's identity in its query string: `?e=you@example.org` plaintext, `?u=`, or `?id=`, which a wordlist or a targeted guess reverses. Publishing that to four third parties is the same leak as publishing the `To` header, hidden one level down. It also **deanonymises the reporter to the attacker**. Abuse desks forward reports to their customers and URLhaus is a public feed, so a kit operator watching for their own URLs learns which target reported them. For a consultant the parameter may carry a CLIENT's identifier rather than the user's own. **So: keep scheme, host and path; keep parameter NAMES; redact parameter VALUES.** ``` http://login-example.invalid/verify?id=REDACTED&src=REDACTED ``` This costs the report almost nothing. Parameter names are part of the kit's fingerprint and campaigns correlate on infrastructure, not on per-victim tokens. The token is unique per recipient BY DESIGN, so it is the one part of the URL that cannot correlate anything, and keeping it makes dedup actively worse: two messages from one campaign would look like different URLs. **Path segments carry the token too.** `/verify/ZGFuaXhAZXhhbXBsZS5vcmc/` is common and a query-string-only rule misses it entirely. A path segment that looks like base64 or a long hex string is FLAGGED FOR REVIEW rather than silently redacted, since a path may also be meaningful. The full URL survives in `source.eml` either way, so the evidence exists locally; it is simply not what gets published by default, and the review gate allows pasting one in by hand when a particular desk genuinely needs it. ## Components ``` abusectl/ cli.py argparse dispatch, exit codes. No logic. case.py case dir: create, load, save manifest, atomic writes parse.py .eml -> IOCs stdlib only, pure contacts.py IOCs -> abuse contacts (RDAP) network, read-only report.py IOCs + contacts -> bodies pure submit.py bodies -> MISP, then vendors network, writes destinations/ one module per target misp.py abusedb.py urlhaus.py virustotal.py email.py config.py ~/.config/abusectl/config.toml ``` **`case.py` is the spine.** Every subcommand goes through it and nothing else writes the case directory. It owns the manifest schema and its version, and it writes ATOMICALLY, temp file plus rename, the way `mailrules.py` saves `rules.json`. A half-written manifest during a review is a corrupted evidence record. **`parse.py` is stdlib-only and pure.** Bytes in, IOC list out; no socket, no config read. That is what makes it testable against fixtures with no setup. **`contacts.py` and `submit.py` are the only network modules**, and both take an INJECTED TRANSPORT. Not an abstraction for its own sake: it is the seam that lets the irreversible path be tested without sending anything. **`destinations/` is one module per target**, each exposing the same two functions, build a body and send one. A fifth vendor is a new file rather than an edit to `submit.py`. This is the one place a plugin shape earns itself, because there are four known members with genuinely different APIs. There is no discovery mechanism; it is a package with four members. **`config.py`** reads TOML through stdlib `tomllib`, no dependency. It holds the cases path, the MISP URL and key, vendor keys, the user's reporting identity for X-ARF, and the trusted-relay boundary described below. Secrets live in a file with mode `0600`, checked on load, and **never reach the manifest or a log**. ### Deliberately absent No database of its own: MISP is the database and case directories are the local record. No daemon, no queue, no retry scheduler, since retry is the user running `submit` again. ### Dependencies by part | Part | Needs | |---|---| | `parse` | stdlib only | | `contacts` | an HTTP client | | `report` | stdlib only | | `submit` | HTTP client, PyMISP | `requirements.txt` is empty until `contacts` is built. **Development runs from a venv in the repository**, gitignored; packaging and distribution are deliberately out of scope until the tool does something worth installing. A SlackBuild in `my-slackbuilds` is the eventual shape, with an nvchecker stanza, matching every other tool the user runs. ## Data flow ``` msg.eml | parse creates case, writes source.eml + iocs[] v case dir --------------------------------------------------+ | contacts RDAP per IOC, writes contacts[] | v | every step case dir | reads and | report writes bodies/, destinations[] | rewrites v | manifest.json case dir -- [ REVIEW: qtmaildir dialog or $EDITOR ] -------+ | submit | +--> MISP ------ fails? STOP, nothing sent --------------+ +--> vendors + abuse desks, per-destination status ``` **Review edits the BODIES, not the IOCs.** Once `report` has run the bodies are the artifact, so `report` refuses to run again on a case whose bodies were modified after generation unless forced, by a timestamp check in the manifest. Silently discarding a review that took twenty minutes is what makes a tool untrustworthy. **`submit` is resumable.** It reads per-destination status and skips anything already `sent`, so running it twice is safe. This matters because the failure that will actually be hit is a rate limit, not a bug. ## Manifest schema ```json { "format": 1, "case_id": "2026-09-08-a3f1", "created": "2026-09-08T12:31:04Z", "source": "source.eml", "iocs": [ { "id": "ioc-1", "type": "ipv4", "value": "203.0.113.42", "origin": "received-chain", "confidence": "untrusted-hop" }, { "id": "ioc-2", "type": "domain", "value": "login-example.invalid", "origin": "href" }, { "id": "ioc-3", "type": "url", "value": "http://login-example.invalid/verify?id=REDACTED", "origin": "href-html" }, { "id": "ioc-4", "type": "sha256", "value": "e3b0c442...", "origin": "attachment", "filename": "invoice.pdf" } ], "auth": { "spf": "fail", "dkim": "none", "dmarc": "fail" }, "contacts": [ { "ioc": "ioc-1", "abuse": "abuse@example.invalid", "source": "rdap", "handle": "AS64496" }, { "ioc": "ioc-2", "abuse": null, "source": "rdap", "error": "no abuse contact published" } ], "destinations": [ { "id": "misp", "kind": "misp", "iocs": ["ioc-1","ioc-2","ioc-3"], "body": null, "status": "pending" }, { "id": "abusedb", "kind": "api", "iocs": ["ioc-1"], "body": "bodies/abusedb.json", "status": "pending" }, { "id": "rdap-1", "kind": "email", "iocs": ["ioc-1"], "target": "abuse@example.invalid", "body": "bodies/rdap-1.xarf", "status": "pending" } ] } ``` After a submit in which one destination failed: ```json { "id": "abusedb", "status": "sent", "sent_at": "2026-09-08T12:40:11Z", "receipt": "8891234" }, { "id": "virustotal", "status": "failed", "attempted_at": "2026-09-08T12:40:12Z", "error": "429 rate limited" } ``` ### Five schema decisions, and why **IOCs carry an `id` and everything references it.** Contacts and destinations point at `ioc-2` rather than repeating the value, so there is one place to correct it and a destination cannot drift from the IOC it reports. **`origin` on every IOC** names where it came from: `received-chain`, `href`, `attachment`. During review the user needs to know whether an IP came from a header to trust or one the attacker wrote. Without it, review is guesswork. **`confidence: untrusted-hop`** exists because of the `Received` trap below. An IP from beneath the trusted-relay boundary is attacker-supplied, and the manifest says so rather than presenting it as fact. **`status` is per destination**, never one status for the case. There is no single answer when four destinations disagree. **MISP is a destination like the others**, with `body: null` because PyMISP builds its own payload. That keeps `submit` one loop with one ordering rule rather than a special case beside a loop. ## Error handling **Partial failure is the normal case, not an exception.** Four destinations can disagree, so each carries its own status and a retry sends only what did not land. All-or-nothing would be wrong here: the successful submissions really happened and cannot be recalled, so recording them as failed would make the next attempt duplicate real reports. **A MISP failure aborts before anything irreversible runs**, per the ordering rule above. **A missing abuse contact is not an error.** RDAP publishes none for many netblocks. The IOC keeps `"abuse": null` with the reason, the destination is simply not created, and review shows what could not be resolved. ## Testing `parse` is fixtures in, JSON out, so the whole reversible half is testable offline with no network and no keys. The network parts get their transport stubbed through the injected seam. **Fixtures use `example.org` and `.invalid` addresses and generic placeholders**, per the user's standing personal-data rule. Real phishing samples do not go in the repository: they carry the recipient identifiers this tool exists to keep out of reports, and a repository is potentially public. Python conventions follow `assets/hooks/` in qtmaildir: GPLv2 header on every file, `unittest`, a `test_.py` beside each module, no framework. ## `parse` in detail: the first part to build Extracted: - **Sending IP** from the `Received` chain: the last UNTRUSTED hop - **Sender domains**: `Return-Path`, `From`, `Reply-To` where it differs - **Href domains and full URLs** from HTML and text parts, decoded, redacted per the rule above, never fetched - **Redirect chains** as DECLARED in headers and href text, never followed - **Auth results**: `Authentication-Results`, and the SPF/DKIM/DMARC verdicts as the receiving server recorded them - **Attachment filenames and SHA-256 hashes**, since a hash is reportable and a filename is an indicator Never captured: recipients, the user's Message-IDs, maildir paths, account keys. ### Two traps, which is where this kind of parser goes wrong **The `Received` chain is attacker-controlled below the user's own infrastructure.** Headers can be forged wholesale, and only the hops the user's own MTA added are trustworthy. Without a configured trusted-relay boundary, the "sending IP" is whatever the attacker chose to write. The boundary is config; IOCs below it are marked `untrusted-hop` rather than presented as fact. **Never resolve and never fetch.** Not the URLs, not the redirects, not remote images. Following a link confirms the address is live to the sender and fires exactly the tracker the message wanted. This is a parser over bytes, and that is a safety property rather than a performance choice. ## The qtmaildir half Specified in item 194 and built after 187 and 190, which settle what Mark spam does and put it on the message bar. It is a review dialog over the case directory: show the bodies, allow editing, show what could not be resolved, and call `submit` on approval, then show per-destination results with a retry for the failures. Sized M in the backlog, not the S a plain button would be: a review dialog is real UI, and the user asked for the review to happen in the application rather than in an editor. The wire contract is this document's manifest plus the `abusectl` command name in qtmaildir's config, in the shape `[sync] command` and the per-account `send_command` already use. ## What each later spec has to settle - **`contacts`**: RDAP bootstrap and referral chasing, caching policy, rate limits, and what happens for a netblock that publishes no abuse contact. - **`report`**: the X-ARF (RFC 5965) schema version, which fields the user's reporting identity fills, and the plain-text alternative for desks that do not parse X-ARF. - **`submit`**: per-vendor auth and payload shapes, MISP event structure (one event per case, or per campaign, and how the dedup query decides), and how mail to an abuse desk is sent, which most likely reuses qtmaildir's own `send_command` rather than adding an SMTP client. - **The qtmaildir dialog**: its own item, after 187 and 190.