path: root/docs/specs/2026-09-08-abusectl-design.md

# abusectl: abuse reporting sidecar, umbrella design

Status: **agreed 2026-09-08**, in one brainstorming session with the user.
This is the UMBRELLA spec. Each part gets its own spec before it is built;
this one settles the decisions the parts share and would be expensive to
change later.

Backlog: item 194 in qtmaildir's
`docs/superpowers/plans/2026-08-03-post-0.1.0-usability.md`, which carries the
qtmaildir half and its ordering against items 187 and 190. This document is
the sidecar, and the two repositories are coupled only by the manifest format
below and the command name in qtmaildir's config.

## Why this exists

The user is a cybersecurity consultant. They have been phished, and they want
to act quickly and effectively on a campaign that targets them: parse a
flagged message, extract its indicators, find who to report each one to, and
file the result both to their own MISP instance and to the public abuse
channels. Filling a database with phishing attempts is a feature no other mail
client offers, which is a reason this one exists.

Two of the user's own constraints are SAFETY PROPERTIES rather than
preferences, stated when the item was first written and not to be traded away
for convenience. They are specified in full below: recipient identifiers are
never captured, and no remote content is ever fetched during parsing.

## Why it is not in qtmaildir

qtmaildir does no network protocol work at all, by design: fetching and
sending are external scripts, which is why there is no IMAP client, no SMTP
client, and why send is a per-account `send_command` on stdin. This tool needs
RDAP, three vendor REST APIs and mail to abuse desks, which is four outbound
protocols. Building it into `src/` would repeal that rule rather than extend
it. Its ecosystem is Python besides, where `assets/hooks/` already puts the
non-GUI parts of this mail system.

**The split is architectural and is not a judgement on the feature.** The
review UI lives in qtmaildir, because a reviewable document is exactly what
makes a GUI review possible.

**A separate repository, not a submodule.** qtmaildir invokes `abusectl` as a
subprocess by name, the way it already invokes `mailsync.sh` and the
`send_command`; nothing in `src/` includes or imports it. A submodule pins a
commit that is not the one that runs, since what executes is whatever is
installed on `$PATH`, and it would cost a two-step clone, an ordering
constraint on every push across two remotes, and an empty directory in the
release tarball, because `git archive` does not include submodule contents.
Compatibility, if it ever matters, is a `format` version in the manifest, the
same discipline `rules.json` already uses across two readers in two languages.

## Shape

One tool, subcommands, one repository. Each subcommand reads and writes a
**case directory**, which is where state lives between steps. A review can
therefore take a week and survive a reboot.

```
abusectl parse msg.eml        -> case dir, IOCs        offline, pure
abusectl contacts <case>      -> + abuse contacts      network, read-only
abusectl report <case>        -> + report bodies       offline, pure
        [ review, in qtmaildir or $EDITOR ]
abusectl submit <case>        -> MISP, then vendors    network, writes
abusectl retry                -> whatever is due       network, writes, cron
```

Every subcommand is independently runnable and independently useful. `parse`
alone triages a message. `parse` + `contacts` + `report` produces a document
that can be sent by hand, with no API key configured anywhere.

**The tool stays usable from a terminal.** The qtmaildir dialog is one front
end, never the only one: a message that never came through qtmaildir, or a
session over SSH, must still be workable. Same reasoning as `mailsync.sh`
printing to stdout as well as its log.

## The one hard ordering rule

`submit` writes **MISP first** and aborts the vendor fan-out if that write
fails.

MISP is the user's own instance and is correctable through its REST API
(update, soft-delete, hard-delete); a submission to AbuseIPDB, URLhaus or
VirusTotal cannot be recalled. So the reversible step gates the irreversible
ones. Reporting externally with no local record of having done so, and no
dedup entry to stop a duplicate later, is worse than not reporting at all.

MISP is also the DEDUP ORACLE. It correlates attributes by value across
events and answers `/attributes/restSearch`, so `submit` can ask whether the
infrastructure has been reported before and skip or attach rather than
duplicate. Writing it before the fan-out is what makes that answer available.

**Correction is MISP's own web UI**, not a subcommand here. A destructive verb
in a tool run often, guarding data the user cares about, is not worth building
for something MISP already does well. A correction subcommand is possible
later.

## The case directory

```
<cases>/2026-09-08-a3f1/
  source.eml          the original, unredacted
  manifest.json       IOCs, contacts, destinations, per-destination status
  bodies/
    abusedb.json
    rdap-1.xarf
```

The path is configurable. **Nothing deletes cases**: they are the user's
evidence, and a tool that silently bins a report sent last month is worse than
a directory that grows. Cleanup is not built.

`manifest.json` carries a `format` version. Bodies are separate files because
they are text a human edits; putting them in the manifest would mean editing
escaped strings inside a JSON document, or growing an extract/apply pair,
which is a directory reinvented with extra steps.

**`source.eml` is unredacted local evidence, and the submit path must never
attach it wholesale.** This is the one route by which the redaction guarantee
below could leak, so it is stated rather than left obvious. The case directory
is sensitive at rest.

## Redaction, structurally

**The parser never captures recipient identifiers.** Not stored, not hashed,
not written to the manifest:

- `To`, `Cc`, `Delivered-To`, `X-Original-To`
- the user's own Message-IDs
- maildir paths and account keys

The guarantee is that **the tool cannot disclose an identifier it was never
given**. It holds no matter what a later subcommand does, which is why it is
enforced at extraction rather than at submission: a strip-on-submit rule
depends on every future sending path remembering, and this one does not.

**The report is reviewable and hand-editable before anything is sent.** What
the tool derives and what the user chooses to disclose are separate things.
The user can add facts during review that the tool never held, including how
many of their addresses a campaign hit, which is a fact they supply from their
own knowledge.

### Tracking parameters are recipient identifiers

A phishing URL commonly carries the recipient's identity in its query string:
`?e=you@example.org` plaintext, `?u=<base64 of the address>`, or
`?id=<md5 of the address>`, which a wordlist or a targeted guess reverses.
Publishing that to four third parties is the same leak as publishing the `To`
header, hidden one level down.

It also **deanonymises the reporter to the attacker**. Abuse desks forward
reports to their customers and URLhaus is a public feed, so a kit operator
watching for their own URLs learns which target reported them. For a
consultant the parameter may carry a CLIENT's identifier rather than the
user's own.

**So: keep scheme, host and path; keep parameter NAMES; redact parameter
VALUES.**

```
http://login-example.invalid/verify?id=REDACTED&src=REDACTED
```

This costs the report almost nothing. Parameter names are part of the kit's
fingerprint and campaigns correlate on infrastructure, not on per-victim
tokens. The token is unique per recipient BY DESIGN, so it is the one part of
the URL that cannot correlate anything, and keeping it makes dedup actively
worse: two messages from one campaign would look like different URLs.

**Path segments carry the token too.** `/verify/ZGFuaXhAZXhhbXBsZS5vcmc/` is
common and a query-string-only rule misses it entirely. A path segment that
looks like base64 or a long hex string is FLAGGED FOR REVIEW rather than
silently redacted, since a path may also be meaningful.

The full URL survives in `source.eml` either way, so the evidence exists
locally; it is simply not what gets published by default, and the review gate
allows pasting one in by hand when a particular desk genuinely needs it.

## Components

```
abusectl/
  cli.py          argparse dispatch, exit codes. No logic.
  case.py         case dir: create, load, save manifest, atomic writes
  parse.py        .eml -> IOCs                       stdlib only, pure
  contacts.py     IOCs -> abuse contacts (RDAP)      network, read-only
  report.py       IOCs + contacts -> bodies          pure
  submit.py       bodies -> MISP, then vendors       network, writes
  retry.py        due destinations, across cases     network, writes
  destinations/   one module per target
    misp.py  abusedb.py  urlhaus.py  virustotal.py  email.py
  config.py       ~/.config/abusectl/config.toml
```

**`case.py` is the spine.** Every subcommand goes through it and nothing else
writes the case directory. It owns the manifest schema and its version, and it
writes ATOMICALLY, temp file plus rename, the way `mailrules.py` saves
`rules.json`. A half-written manifest during a review is a corrupted evidence
record.

**`parse.py` is stdlib-only and pure.** Bytes in, IOC list out; no socket, no
config read. That is what makes it testable against fixtures with no setup.

**`contacts.py`, `submit.py` and `retry.py` are the only network modules**,
and each takes an INJECTED TRANSPORT. Not an abstraction for its own sake: it is the seam
that lets the irreversible path be tested without sending anything.

**`destinations/` is one module per target**, each exposing the same two
functions, build a body and send one. A fifth vendor is a new file rather than
an edit to `submit.py`. This is the one place a plugin shape earns itself,
because there are four known members with genuinely different APIs. There is
no discovery mechanism; it is a package with four members.

**`config.py`** reads TOML through stdlib `tomllib`, no dependency. It holds
the cases path, the MISP URL and key, vendor keys, the user's reporting
identity for X-ARF, `max_attempts` for the retry cap, and the trusted-relay
boundary described below. Secrets
live in a file with mode `0600`, checked on load, and **never reach the
manifest or a log**.

### Deliberately absent

No database of its own: MISP is the database and case directories are the
local record. **No daemon and no queue.** Retry is a `retry` subcommand run
from cron, described below; the scheduling belongs to cron, which the user
already runs `mailsync.sh` from, and not to a process of this tool's own.

### Dependencies by part

| Part | Needs |
|---|---|
| `parse` | stdlib only |
| `contacts` | an HTTP client |
| `report` | stdlib only |
| `submit` | HTTP client, PyMISP |
| `retry` | HTTP client (no PyMISP: it never writes MISP) |

`requirements.txt` is empty until `contacts` is built. **Development runs from
a venv in the repository**, gitignored; packaging and distribution are
deliberately out of scope until the tool does something worth installing. A
SlackBuild in `my-slackbuilds` is the eventual shape, with an nvchecker
stanza, matching every other tool the user runs.

## Data flow

```
msg.eml
   |  parse      creates case, writes source.eml + iocs[]
   v
case dir --------------------------------------------------+
   |  contacts   RDAP per IOC, writes contacts[]            |
   v                                                        |  every step
case dir                                                    |  reads and
   |  report     writes bodies/, destinations[]             |  rewrites
   v                                                        |  manifest.json
case dir  -- [ REVIEW: qtmaildir dialog or $EDITOR ] -------+
   |  submit                                                |
   +--> MISP ------ fails? STOP, nothing sent --------------+
   +--> vendors + abuse desks, per-destination status
             |
             +--> rate limited? status `deferred` + retry_after
                        |
                        v
                  cron: abusectl retry     every N minutes,
                  sends only what is due   across every case
```

**Review edits the BODIES, not the IOCs.** Once `report` has run the bodies
are the artifact, so `report` refuses to run again on a case whose bodies were
modified after generation unless forced, by a timestamp check in the manifest.
Silently discarding a review that took twenty minutes is what makes a tool
untrustworthy.

**`submit` is resumable.** It reads per-destination status and skips anything
already `sent`, so running it twice is safe. This matters because the failure
that will actually be hit is a rate limit, not a bug, and because `retry`
below runs unattended over the same cases.

## Manifest schema

```json
{
  "format": 1,
  "case_id": "2026-09-08-a3f1",
  "created": "2026-09-08T12:31:04Z",
  "source": "source.eml",

  "iocs": [
    { "id": "ioc-1", "type": "ipv4", "value": "203.0.113.42",
      "origin": "received-chain", "confidence": "untrusted-hop" },
    { "id": "ioc-2", "type": "domain", "value": "login-example.invalid",
      "origin": "href" },
    { "id": "ioc-3", "type": "url",
      "value": "http://login-example.invalid/verify?id=REDACTED",
      "origin": "href-html" },
    { "id": "ioc-4", "type": "sha256", "value": "e3b0c442...",
      "origin": "attachment", "filename": "invoice.pdf" }
  ],

  "auth": { "spf": "fail", "dkim": "none", "dmarc": "fail" },

  "contacts": [
    { "ioc": "ioc-1", "abuse": "abuse@example.invalid",
      "source": "rdap", "handle": "AS64496" },
    { "ioc": "ioc-2", "abuse": null, "source": "rdap",
      "error": "no abuse contact published" }
  ],

  "destinations": [
    { "id": "misp",    "kind": "misp",  "iocs": ["ioc-1","ioc-2","ioc-3"],
      "body": null,                    "status": "pending" },
    { "id": "abusedb", "kind": "api",   "iocs": ["ioc-1"],
      "body": "bodies/abusedb.json",   "status": "pending" },
    { "id": "rdap-1",  "kind": "email", "iocs": ["ioc-1"],
      "target": "abuse@example.invalid",
      "body": "bodies/rdap-1.xarf",    "status": "pending" }
  ]
}
```

After a submit in which one destination landed, one was rate limited and one
is genuinely broken:

```json
{ "id": "abusedb", "status": "sent",
  "sent_at": "2026-09-08T12:40:11Z", "receipt": "8891234" },

{ "id": "virustotal", "status": "deferred",
  "attempted_at": "2026-09-08T12:40:12Z",
  "retry_after": "2026-09-08T13:00:00Z",
  "attempts": 1, "error": "429 rate limited" },

{ "id": "rdap-1", "status": "failed",
  "attempted_at": "2026-09-08T12:40:13Z",
  "attempts": 5, "error": "550 no such mailbox" }
```

### The five statuses

| status | meaning | who acts |
|---|---|---|
| `pending` | never attempted | `submit` |
| `in-flight` | attempt started, outcome unknown | a human |
| `sent` | landed, receipt recorded | nobody, ever again |
| `deferred` | soft failure, `retry_after` set | `retry`, from cron |
| `failed` | hard failure, or attempts exhausted | a human |

`deferred` and `failed` are deliberately distinct. Deferred means the tool
will handle it; failed means it will not, and the user has to look. Collapsing
them either strands a rate-limited report forever or retries a dead mailbox
until the end of time.

### Five schema decisions, and why

**IOCs carry an `id` and everything references it.** Contacts and destinations
point at `ioc-2` rather than repeating the value, so there is one place to
correct it and a destination cannot drift from the IOC it reports.

**`origin` on every IOC** names where it came from: `received-chain`, `href`,
`attachment`. During review the user needs to know whether an IP came from a
header to trust or one the attacker wrote. Without it, review is guesswork.

**`confidence: untrusted-hop`** exists because of the `Received` trap below.
An IP from beneath the trusted-relay boundary is attacker-supplied, and the
manifest says so rather than presenting it as fact.

**`status` is per destination**, never one status for the case. There is no
single answer when four destinations disagree.

**MISP is a destination like the others**, with `body: null` because PyMISP
builds its own payload. That keeps `submit` one loop with one ordering rule
rather than a special case beside a loop.

## Error handling

**Partial failure is the normal case, not an exception.** Four destinations
can disagree, so each carries its own status and a retry sends only what did
not land. All-or-nothing would be wrong here: the successful submissions
really happened and cannot be recalled, so recording them as failed would make
the next attempt duplicate real reports.

**A MISP failure aborts before anything irreversible runs**, per the ordering
rule above.

**A missing abuse contact is not an error.** RDAP publishes none for many
netblocks. The IOC keeps `"abuse": null` with the reason, the destination is
simply not created, and review shows what could not be resolved.

## Retry, from cron

A rate limit tells us WHEN, not merely that: a 429 carries `Retry-After`, or
the vendor's rate-limit reset headers do. Discarding that and asking the user
to run `submit` again throws away a fact the server handed us, and relies on
them remembering. So the deadline is recorded and cron acts on it.

```
*/N * * * *  abusectl retry
```

`abusectl retry` scans every case under the configured cases directory, finds
destinations whose status is `deferred` and whose `retry_after` has passed,
and sends only those. It takes no case argument: the point is that it needs no
knowledge of what is outstanding, so it can be a single unattended cron line
beside `mailsync.sh`.

**There is no inline retry.** `submit` never sleeps waiting for a window to
open. A foreground process blocked for an unknown period is worse than the
problem it solves, since a daily quota resets in hours rather than seconds,
and a process killed while sleeping is back to the user remembering. One
mechanism, not two.

### What makes unattended retry safe

Automated retry turns three properties from conveniences into load-bearing
ones. A retry that re-sends is a duplicate abuse report, which cannot be
withdrawn.

**Status is written BEFORE the attempt, not only after.** A destination is
marked `in-flight` with its manifest flushed to disk, then attempted, then
marked `sent` or `deferred`. A crash between the send and the status write
therefore leaves `in-flight`, which is honest: the report may or may not have
arrived. **`retry` never touches an `in-flight` destination**; it is for a
human to resolve by checking the receiving end. The alternative, writing only
after, makes an interrupted send indistinguishable from one that never
happened, and cron would re-send it.

**`attempts` is counted and capped.** A destination that has failed
`max_attempts` times (config, default 5) becomes `failed` rather than
`deferred`, so a permanently broken abuse mailbox is not retried forever. The
count is per destination, not per case.

**Backoff on a deadline the server did not give.** Where a failure is soft but
carries no `Retry-After`, `retry_after` is set by exponential backoff from
`attempts`, so a flapping endpoint is not hammered every N minutes.

### What retry must never do

- **Never re-send a `sent` destination.** The receipt is recorded; there is
  nothing to redo.
- **Never touch `in-flight`.** Ambiguous is not the same as failed.
- **Never run the MISP ordering rule again.** MISP was written before the
  fan-out during the original `submit`, so by the time a destination is
  `deferred` the local record already exists. `retry` sends vendor and email
  destinations only; a MISP write that failed aborted the submission before
  any of them was attempted, so there is nothing deferred to pick up.
- **Never re-read or re-generate the bodies.** It sends what was reviewed. A
  body edited after submission is not silently picked up, for the same reason
  `report` refuses to overwrite a reviewed case.

### Concurrency

`retry` runs from cron and the user may run `submit` by hand at the same
moment. Each case directory is locked for the duration of an attempt, with a
lock file inside it, and a case that is locked is skipped rather than waited
for: the next cron tick will reach it. This is the same reasoning as
`mailsync.sh` exiting 75 when another run holds its lock.

## Testing

`parse` is fixtures in, JSON out, so the whole reversible half is testable
offline with no network and no keys. The network parts get their transport
stubbed through the injected seam.

**Fixtures use `example.org` and `.invalid` addresses and generic
placeholders**, per the user's standing personal-data rule. Real phishing
samples do not go in the repository: they carry the recipient identifiers this
tool exists to keep out of reports, and a repository is potentially public.

Python conventions follow `assets/hooks/` in qtmaildir: GPLv2 header on every
file, `unittest`, a `test_<module>.py` beside each module, no framework.

## `parse` in detail: the first part to build

Extracted:

- **Sending IP** from the `Received` chain: the last UNTRUSTED hop
- **Sender domains**: `Return-Path`, `From`, `Reply-To` where it differs
- **Href domains and full URLs** from HTML and text parts, decoded, redacted
  per the rule above, never fetched
- **Redirect chains** as DECLARED in headers and href text, never followed
- **Auth results**: `Authentication-Results`, and the SPF/DKIM/DMARC verdicts
  as the receiving server recorded them
- **Attachment filenames and SHA-256 hashes**, since a hash is reportable and
  a filename is an indicator

Never captured: recipients, the user's Message-IDs, maildir paths, account
keys.

### Two traps, which is where this kind of parser goes wrong

**The `Received` chain is attacker-controlled below the user's own
infrastructure.** Headers can be forged wholesale, and only the hops the
user's own MTA added are trustworthy. Without a configured trusted-relay
boundary, the "sending IP" is whatever the attacker chose to write. The
boundary is config; IOCs below it are marked `untrusted-hop` rather than
presented as fact.

**Never resolve and never fetch.** Not the URLs, not the redirects, not remote
images. Following a link confirms the address is live to the sender and fires
exactly the tracker the message wanted. This is a parser over bytes, and that
is a safety property rather than a performance choice.

## The qtmaildir half

Specified in item 194 and built after 187 and 190, which settle what Mark spam
does and put it on the message bar. It is a review dialog over the case
directory: show the bodies, allow editing, show what could not be resolved,
and call `submit` on approval, then show per-destination results.

It offers no retry button. A `deferred` destination belongs to cron, and a
button beside it would race the scheduled run and risk a duplicate report; the
dialog says when the retry is due instead. A `failed` or `in-flight`
destination needs the user to act somewhere else, not to press a button that
would repeat what already did not work. Sized M in the backlog, not the S a plain button would be:
a review dialog is real UI, and the user asked for the review to happen in the
application rather than in an editor.

The wire contract is this document's manifest plus the `abusectl` command name
in qtmaildir's config, in the shape `[sync] command` and the per-account
`send_command` already use.

## What each later spec has to settle

- **`contacts`**: RDAP bootstrap and referral chasing, caching policy, rate
  limits, and what happens for a netblock that publishes no abuse contact.
- **`report`**: the X-ARF (RFC 5965) schema version, which fields the user's
  reporting identity fills, and the plain-text alternative for desks that do
  not parse X-ARF.
- **`submit` and `retry`**: per-vendor auth and payload shapes, which
  rate-limit headers each vendor actually sends and how a soft failure is
  told from a hard one, the backoff curve, MISP event structure
  (one event per case, or per campaign, and how the dedup query decides),
  and how mail to an abuse desk is sent, which most likely reuses qtmaildir's
  own `send_command` rather than adding an SMTP client.
- **The qtmaildir dialog**: its own item, after 187 and 190.