path: root/docs/specs/2026-09-08-abusectl-design.md

# abusectl: abuse reporting sidecar, umbrella design

Status: **agreed 2026-09-08**, in one brainstorming session with the user.
This is the UMBRELLA spec. Each part gets its own spec before it is built;
this one settles the decisions the parts share and would be expensive to
change later.

Backlog: item 194 in qtmaildir's
`docs/superpowers/plans/2026-08-03-post-0.1.0-usability.md`, which carries the
qtmaildir half and its ordering against items 187 and 190. This document is
the sidecar, and the two repositories are coupled only by the manifest format
below and the command name in qtmaildir's config.

## Why this exists

The user is a cybersecurity consultant. They have been phished, and they want
to act quickly and effectively on a campaign that targets them: parse a
flagged message, extract its indicators, find who to report each one to, and
file the result both to their own MISP instance and to the public abuse
channels. Filling a database with phishing attempts is a feature no other mail
client offers, which is a reason this one exists.

Two of the user's own constraints are SAFETY PROPERTIES rather than
preferences, stated when the item was first written and not to be traded away
for convenience. They are specified in full below: recipient identifiers are
never captured, and no remote content is ever fetched during parsing.

## Why it is not in qtmaildir

qtmaildir does no network protocol work at all, by design: fetching and
sending are external scripts, which is why there is no IMAP client, no SMTP
client, and why send is a per-account `send_command` on stdin. This tool needs
RDAP, three vendor REST APIs and mail to abuse desks, which is four outbound
protocols. Building it into `src/` would repeal that rule rather than extend
it. Its ecosystem is Python besides, where `assets/hooks/` already puts the
non-GUI parts of this mail system.

**The split is architectural and is not a judgement on the feature.** The
review UI lives in qtmaildir, because a reviewable document is exactly what
makes a GUI review possible.

**A separate repository, not a submodule.** qtmaildir invokes `abusectl` as a
subprocess by name, the way it already invokes `mailsync.sh` and the
`send_command`; nothing in `src/` includes or imports it. A submodule pins a
commit that is not the one that runs, since what executes is whatever is
installed on `$PATH`, and it would cost a two-step clone, an ordering
constraint on every push across two remotes, and an empty directory in the
release tarball, because `git archive` does not include submodule contents.
Compatibility, if it ever matters, is a `format` version in the manifest, the
same discipline `rules.json` already uses across two readers in two languages.

## Shape

One tool, subcommands, one repository. Each subcommand reads and writes a
**case directory**, which is where state lives between steps. A review can
therefore take a week and survive a reboot.

```
abusectl init                 -> config.toml          first run, then exits
abusectl parse msg.eml        -> case dir, IOCs        offline, pure
abusectl contacts <case>      -> + abuse contacts      network, read-only
abusectl report <case>        -> + report bodies       offline, pure
        [ review, in qtmaildir or $EDITOR ]
abusectl submit <case>        -> MISP, then vendors    network, writes
abusectl retry                -> whatever is due       network, writes, cron
```

Every subcommand is independently runnable and independently useful. `parse`
alone triages a message. `parse` + `contacts` + `report` produces a document
that can be sent by hand, with no API key configured anywhere.

**The tool stays usable from a terminal.** The qtmaildir dialog is one front
end, never the only one: a message that never came through qtmaildir, or a
session over SSH, must still be workable. Same reasoning as `mailsync.sh`
printing to stdout as well as its log.

## The one hard ordering rule

`submit` writes **MISP first** and aborts the vendor fan-out if that write
fails.

MISP is the user's own instance and is correctable through its REST API
(update, soft-delete, hard-delete); a submission to AbuseIPDB, URLhaus or
VirusTotal cannot be recalled. So the reversible step gates the irreversible
ones. Reporting externally with no local record of having done so, and no
dedup entry to stop a duplicate later, is worse than not reporting at all.

MISP is also the DEDUP ORACLE. It correlates attributes by value across
events and answers `/attributes/restSearch`, so `submit` can ask whether the
infrastructure has been reported before and skip or attach rather than
duplicate. Writing it before the fan-out is what makes that answer available.

**Correction is MISP's own web UI**, not a subcommand here. A destructive verb
in a tool run often, guarding data the user cares about, is not worth building
for something MISP already does well. A correction subcommand is possible
later.

## The case directory

```
<cases>/2026-09-08-a3f1/
  source.eml          the original, unredacted
  manifest.json       IOCs, contacts, destinations, per-destination status
  bodies/
    abusedb.json
    rdap-1.xarf
```

The path is configurable. **Nothing deletes cases**: they are the user's
evidence, and a tool that silently bins a report sent last month is worse than
a directory that grows. Cleanup is not built.

`manifest.json` carries a `format` version. Bodies are separate files because
they are text a human edits; putting them in the manifest would mean editing
escaped strings inside a JSON document, or growing an extract/apply pair,
which is a directory reinvented with extra steps.

**`source.eml` is unredacted local evidence, and the submit path must never
attach it wholesale.** This is the one route by which the redaction guarantee
below could leak, so it is stated rather than left obvious. The case directory
is sensitive at rest.

## Redaction, structurally

**The parser never captures recipient identifiers.** Not stored, not hashed,
not written to the manifest:

- `To`, `Cc`, `Delivered-To`, `X-Original-To`
- the user's own Message-IDs
- maildir paths and account keys

The guarantee is that **the tool cannot disclose an identifier it was never
given**. It holds no matter what a later subcommand does, which is why it is
enforced at extraction rather than at submission: a strip-on-submit rule
depends on every future sending path remembering, and this one does not.

**The report is reviewable and hand-editable before anything is sent.** What
the tool derives and what the user chooses to disclose are separate things.
The user can add facts during review that the tool never held, including how
many of their addresses a campaign hit, which is a fact they supply from their
own knowledge.

### Tracking parameters are recipient identifiers

A phishing URL commonly carries the recipient's identity in its query string:
`?e=you@example.org` plaintext, `?u=<base64 of the address>`, or
`?id=<md5 of the address>`, which a wordlist or a targeted guess reverses.
Publishing that to four third parties is the same leak as publishing the `To`
header, hidden one level down.

It also **deanonymises the reporter to the attacker**. Abuse desks forward
reports to their customers and URLhaus is a public feed, so a kit operator
watching for their own URLs learns which target reported them. For a
consultant the parameter may carry a CLIENT's identifier rather than the
user's own.

**So: keep scheme, host and path; keep parameter NAMES; redact parameter
VALUES.**

```
http://login-example.invalid/verify?id=REDACTED&src=REDACTED
```

This costs the report almost nothing. Parameter names are part of the kit's
fingerprint and campaigns correlate on infrastructure, not on per-victim
tokens. The token is unique per recipient BY DESIGN, so it is the one part of
the URL that cannot correlate anything, and keeping it makes dedup actively
worse: two messages from one campaign would look like different URLs.

**Path segments carry the token too.** `/verify/ZGFuaXhAZXhhbXBsZS5vcmc/` is
common and a query-string-only rule misses it entirely. A path segment that
looks like base64 or a long hex string is FLAGGED FOR REVIEW rather than
silently redacted, since a path may also be meaningful.

The full URL survives in `source.eml` either way, so the evidence exists
locally; it is simply not what gets published by default, and the review gate
allows pasting one in by hand when a particular desk genuinely needs it.

**One exception, and it is the rule's own logic rather than a hole in it.** A
redirector carries its DESTINATION in a parameter, which the rule above would
blank. That destination is an indicator rather than a recipient identifier, so
a parameter value that parses as an http(s) URL is recovered and reported in
its own right, redacted itself, and the hop is recorded as a chain. Every
value that is not a URL stays blanked, including a tracking token sitting in
the same query string. The recovery is bounded in depth, because a redirector
may point at another one and the nested value is attacker-supplied.

Reading a destination out of a parameter is not fetching it. The chain is what
the message DECLARES; nothing is followed.

## Components

```
abusectl/
  cli.py          argparse dispatch, exit codes. No logic.
  init.py         first-run config: pure builder + prompt shell
  case.py         case dir: create, load, save manifest, atomic writes
  parse.py        .eml -> IOCs                       stdlib only, pure
  contacts.py     IOCs -> abuse contacts (RDAP)      network, read-only
  report.py       IOCs + contacts -> bodies          pure
  submit.py       bodies -> MISP, then vendors       network, writes
  retry.py        due destinations, across cases     network, writes
  destinations/   one module per target
    misp.py  abusedb.py  urlhaus.py  virustotal.py  email.py
  config.py       ~/.config/abusectl/config.toml
```

**`case.py` is the spine.** Every subcommand goes through it and nothing else
writes the case directory. It owns the manifest schema and its version, and it
writes ATOMICALLY, temp file plus rename, the way `mailrules.py` saves
`rules.json`. A half-written manifest during a review is a corrupted evidence
record.

**`parse.py` is stdlib-only and pure.** Bytes in, IOC list out; no socket, no
config read. That is what makes it testable against fixtures with no setup.

**`contacts.py`, `submit.py` and `retry.py` are the only network modules**,
and each takes an INJECTED TRANSPORT. Not an abstraction for its own sake: it is the seam
that lets the irreversible path be tested without sending anything.

**`destinations/` is one module per target**, each exposing the same two
functions, build a body and send one. A fifth vendor is a new file rather than
an edit to `submit.py`. This is the one place a plugin shape earns itself,
because there are four known members with genuinely different APIs. There is
no discovery mechanism; it is a package with four members.

## First run: `abusectl init`

**`parse` REFUSES to run with no `trusted_relays` configured**, rather than
guessing. The outermost public IP is the usual guess and it is wrong in
exactly the case that matters: an attacker who forges extra `Received`
headers. A confident wrong answer here gets an innocent third party reported.

Refusing is only defensible with a route out, so `abusectl init` writes the
config and exits, and `parse`'s error names it rather than stating a bare
failure.

**It asks only what the part being built needs.** Today that is the trusted
relays and the cases directory. The MISP URL and key, the vendor keys and the
X-ARF identity arrive as questions when the parts that use them are built, so
no question is written before its part is designed.

**A skipped answer is ABSENT from the file, never an empty string.**
`api_key = ""` reads as configured-and-broken and produces a confusing auth
error much later; an absent key reads as not-configured, and the part that
wants it can say so plainly.

### The trusted-relay question has three tiers

It is the answer a user is most likely to get wrong, and it is the one that
decides whether the reported IP is the attacker's or an innocent relay's.

1. **Ask for CIDRs.** Validated with stdlib `ipaddress`, so a malformed entry
   is rejected at the prompt rather than at parse time.
2. **A known-provider table.** Gmail, Fastmail, Proton and similar publish
   their sending ranges. The table ships with the tool as static data.
   Deliberately NOT read from SPF at runtime: that is a DNS lookup, and while
   the never-resolve rule is about parsing hostile mail rather than about
   setup, a static table keeps the boundary unambiguous.
3. **`--from-sample <msg.eml>`.** For self-hosted mail and anything absent
   from the table: show the `Received` chain of a message the user knows
   arrived legitimately, and let them pick which hops are theirs. This turns
   an abstract question into choosing from a real list.

### Two front ends, one writer

`init` is a **pure builder plus a thin prompt shell**. One function takes the
answers as a mapping and returns the TOML text; the interactive prompts and
the command-line flags are two front ends over it. So the config-writing logic
is testable with no terminal and no files, and the two routes cannot drift.

**`--non-interactive` exists so an agent can run setup**, taking every answer
as a flag and failing on a missing one rather than prompting. Every question
is also a flag; there is no answer reachable only by typing.

**`init` never overwrites silently.** Re-running over an existing config
refuses unless `--force`, and `--force` still never drops a key the new run
did not supply: a config holding a MISP key must not lose it to a later run
that only set the relays.

### What is tested and what is hand-tested

The builder has right answers and is tested: the TOML it produces, a skipped
key being absent, a malformed CIDR being rejected, an existing config not
being clobbered. **The prompts are hand-tested by the user**, because whether
a question reads clearly has no assertion; a test driving stdin would assert
the wording it was written against and break on a rewording that improved it.

**`config.py`** reads TOML through stdlib `tomllib`, no dependency. It holds
the cases path, the MISP URL and key, vendor keys, the user's reporting
identity for X-ARF, `max_attempts` for the retry cap, and the trusted-relay
boundary described below. Secrets
live in a file with mode `0600`, checked on load, and **never reach the
manifest or a log**.

### Deliberately absent

No database of its own: MISP is the database and case directories are the
local record. **No daemon and no queue.** Retry is a `retry` subcommand run
from cron, described below; the scheduling belongs to cron, which the user
already runs `mailsync.sh` from, and not to a process of this tool's own.

### Dependencies by part

| Part | Needs |
|---|---|
| `init` | stdlib only |
| `parse` | stdlib only |
| `contacts` | an HTTP client |
| `report` | stdlib only |
| `submit` | HTTP client, PyMISP |
| `retry` | HTTP client (no PyMISP: it never writes MISP) |

`requirements.txt` is empty until `contacts` is built. **Development runs from
a venv in the repository**, gitignored; packaging and distribution are
deliberately out of scope until the tool does something worth installing. A
SlackBuild in `my-slackbuilds` is the eventual shape, with an nvchecker
stanza, matching every other tool the user runs.

## Data flow

```
msg.eml
   |  parse      creates case, writes source.eml + iocs[]
   v
case dir --------------------------------------------------+
   |  contacts   RDAP per IOC, writes contacts[]            |
   v                                                        |  every step
case dir                                                    |  reads and
   |  report     writes bodies/, destinations[]             |  rewrites
   v                                                        |  manifest.json
case dir  -- [ REVIEW: qtmaildir dialog or $EDITOR ] -------+
   |  submit                                                |
   +--> MISP ------ fails? STOP, nothing sent --------------+
   +--> vendors + abuse desks, per-destination status
             |
             +--> rate limited? status `deferred` + retry_after
                        |
                        v
                  cron: abusectl retry     every N minutes,
                  sends only what is due   across every case
```

**Review edits the BODIES, not the IOCs.** Once `report` has run the bodies
are the artifact, so `report` refuses to run again on a case whose bodies were
modified after generation unless forced, by a timestamp check in the manifest.
Silently discarding a review that took twenty minutes is what makes a tool
untrustworthy.

**`submit` is resumable.** It reads per-destination status and skips anything
already `sent`, so running it twice is safe. This matters because the failure
that will actually be hit is a rate limit, not a bug, and because `retry`
below runs unattended over the same cases.

## Manifest schema

```json
{
  "format": 1,
  "case_id": "2026-09-08-a3f1",
  "created": "2026-09-08T12:31:04Z",
  "source": "source.eml",

  "iocs": [
    { "id": "ioc-1", "type": "ipv4", "value": "203.0.113.42",
      "origin": "received-chain", "confidence": "untrusted-hop" },
    { "id": "ioc-2", "type": "domain", "value": "login-example.invalid",
      "origin": "href" },
    { "id": "ioc-3", "type": "url",
      "value": "http://login-example.invalid/verify?id=REDACTED",
      "origin": "href-html" },
    { "id": "ioc-4", "type": "sha256", "value": "e3b0c442...",
      "origin": "attachment", "filename": "invoice.pdf" }
  ],

  "auth": { "spf": "fail", "dkim": "none", "dmarc": "fail" },

  "contacts": [
    { "ioc": "ioc-1", "abuse": "abuse@example.invalid",
      "source": "rdap", "handle": "AS64496" },
    { "ioc": "ioc-2", "abuse": null, "source": "rdap",
      "error": "no abuse contact published" }
  ],

  "destinations": [
    { "id": "misp",    "kind": "misp",  "iocs": ["ioc-1","ioc-2","ioc-3"],
      "body": null,                    "status": "pending" },
    { "id": "abusedb", "kind": "api",   "iocs": ["ioc-1"],
      "body": "bodies/abusedb.json",   "status": "pending" },
    { "id": "rdap-1",  "kind": "email", "iocs": ["ioc-1"],
      "target": "abuse@example.invalid",
      "body": "bodies/rdap-1.xarf",    "status": "pending" }
  ]
}
```

After a submit in which one destination landed, one was rate limited and one
is genuinely broken:

```json
{ "id": "abusedb", "status": "sent",
  "sent_at": "2026-09-08T12:40:11Z", "receipt": "8891234" },

{ "id": "virustotal", "status": "deferred",
  "attempted_at": "2026-09-08T12:40:12Z",
  "retry_after": "2026-09-08T13:00:00Z",
  "attempts": 1, "error": "429 rate limited" },

{ "id": "rdap-1", "status": "failed",
  "attempted_at": "2026-09-08T12:40:13Z",
  "attempts": 5, "error": "550 no such mailbox" }
```

### The five statuses

| status | meaning | who acts |
|---|---|---|
| `pending` | never attempted | `submit` |
| `in-flight` | attempt started, outcome unknown | a human |
| `sent` | landed, receipt recorded | nobody, ever again |
| `deferred` | soft failure, `retry_after` set | `retry`, from cron |
| `failed` | hard failure, or attempts exhausted | a human |

`deferred` and `failed` are deliberately distinct. Deferred means the tool
will handle it; failed means it will not, and the user has to look. Collapsing
them either strands a rate-limited report forever or retries a dead mailbox
until the end of time.

### Five schema decisions, and why

**IOCs carry an `id` and everything references it.** Contacts and destinations
point at `ioc-2` rather than repeating the value, so there is one place to
correct it and a destination cannot drift from the IOC it reports.

**`origin` on every IOC** names where it came from: `received-chain`, `href`,
`attachment`. During review the user needs to know whether an IP came from a
header to trust or one the attacker wrote. Without it, review is guesswork.

**`confidence: untrusted-hop`** exists because of the `Received` trap below.
An IP from beneath the trusted-relay boundary is attacker-supplied, and the
manifest says so rather than presenting it as fact.

**`status` is per destination**, never one status for the case. There is no
single answer when four destinations disagree.

**MISP is a destination like the others**, with `body: null` because PyMISP
builds its own payload. That keeps `submit` one loop with one ordering rule
rather than a special case beside a loop.

## Error handling

**Partial failure is the normal case, not an exception.** Four destinations
can disagree, so each carries its own status and a retry sends only what did
not land. All-or-nothing would be wrong here: the successful submissions
really happened and cannot be recalled, so recording them as failed would make
the next attempt duplicate real reports.

**A MISP failure aborts before anything irreversible runs**, per the ordering
rule above.

**A missing abuse contact is not an error.** RDAP publishes none for many
netblocks. The IOC keeps `"abuse": null` with the reason, the destination is
simply not created, and review shows what could not be resolved.

## Retry, from cron

A rate limit tells us WHEN, not merely that: a 429 carries `Retry-After`, or
the vendor's rate-limit reset headers do. Discarding that and asking the user
to run `submit` again throws away a fact the server handed us, and relies on
them remembering. So the deadline is recorded and cron acts on it.

```
*/N * * * *  abusectl retry
```

`abusectl retry` scans every case under the configured cases directory, finds
destinations whose status is `deferred` and whose `retry_after` has passed,
and sends only those. It takes no case argument: the point is that it needs no
knowledge of what is outstanding, so it can be a single unattended cron line
beside `mailsync.sh`.

**There is no inline retry.** `submit` never sleeps waiting for a window to
open. A foreground process blocked for an unknown period is worse than the
problem it solves, since a daily quota resets in hours rather than seconds,
and a process killed while sleeping is back to the user remembering. One
mechanism, not two.

### What makes unattended retry safe

Automated retry turns three properties from conveniences into load-bearing
ones. A retry that re-sends is a duplicate abuse report, which cannot be
withdrawn.

**Status is written BEFORE the attempt, not only after.** A destination is
marked `in-flight` with its manifest flushed to disk, then attempted, then
marked `sent` or `deferred`. A crash between the send and the status write
therefore leaves `in-flight`, which is honest: the report may or may not have
arrived. **`retry` never touches an `in-flight` destination**; it is for a
human to resolve by checking the receiving end. The alternative, writing only
after, makes an interrupted send indistinguishable from one that never
happened, and cron would re-send it.

**`attempts` is counted and capped.** A destination that has failed
`max_attempts` times (config, default 5) becomes `failed` rather than
`deferred`, so a permanently broken abuse mailbox is not retried forever. The
count is per destination, not per case.

**Backoff on a deadline the server did not give.** Where a failure is soft but
carries no `Retry-After`, `retry_after` is set by exponential backoff from
`attempts`, so a flapping endpoint is not hammered every N minutes.

### What retry must never do

- **Never re-send a `sent` destination.** The receipt is recorded; there is
  nothing to redo.
- **Never touch `in-flight`.** Ambiguous is not the same as failed.
- **Never run the MISP ordering rule again.** MISP was written before the
  fan-out during the original `submit`, so by the time a destination is
  `deferred` the local record already exists. `retry` sends vendor and email
  destinations only; a MISP write that failed aborted the submission before
  any of them was attempted, so there is nothing deferred to pick up.
- **Never re-read or re-generate the bodies.** It sends what was reviewed. A
  body edited after submission is not silently picked up, for the same reason
  `report` refuses to overwrite a reviewed case.

### Concurrency

`retry` runs from cron and the user may run `submit` by hand at the same
moment. Each case directory is locked for the duration of an attempt, with a
lock file inside it, and a case that is locked is skipped rather than waited
for: the next cron tick will reach it. This is the same reasoning as
`mailsync.sh` exiting 75 when another run holds its lock.

## Testing

`parse` is fixtures in, JSON out, so the whole reversible half is testable
offline with no network and no keys. The network parts get their transport
stubbed through the injected seam.

**Fixtures use `example.org` and `.invalid` addresses and generic
placeholders**, per the user's standing personal-data rule. Real phishing
samples do not go in the repository: they carry the recipient identifiers this
tool exists to keep out of reports, and a repository is potentially public.

Python conventions follow `assets/hooks/` in qtmaildir: GPLv2 header on every
file, `unittest`, a `test_<module>.py` beside each module, no framework.

## `parse` in detail: the first part to build

Extracted:

- **Sending IP** from the `Received` chain: the last UNTRUSTED hop
- **Sender domains**: `Return-Path`, `From`, `Reply-To` where it differs
- **Href domains and full URLs** from HTML and text parts, decoded, redacted
  per the rule above, never fetched
- **Redirect chains** as DECLARED in headers and href text, never followed
- **Auth results**: `Authentication-Results`, and the SPF/DKIM/DMARC verdicts
  as the receiving server recorded them
- **Attachment filenames and SHA-256 hashes**, since a hash is reportable and
  a filename is an indicator

Never captured: recipients, the user's Message-IDs, maildir paths, account
keys.

### Two traps, which is where this kind of parser goes wrong

**The `Received` chain is attacker-controlled below the user's own
infrastructure.** Headers can be forged wholesale, and only the hops the
user's own MTA added are trustworthy. Without a configured trusted-relay
boundary, the "sending IP" is whatever the attacker chose to write. The
boundary is config, supplied to `parse` as an ARGUMENT so the module stays
pure and config-free; `cli.py` reads it and passes it in. IOCs below the
boundary are marked `untrusted-hop` rather than presented as fact, and with no
boundary configured at all `parse` refuses to run rather than guessing, per
the `init` section above.

**Never resolve and never fetch.** Not the URLs, not the redirects, not remote
images. Following a link confirms the address is live to the sender and fires
exactly the tracker the message wanted. This is a parser over bytes, and that
is a safety property rather than a performance choice.

## The qtmaildir half

Specified in item 194 and built after 187 and 190, which settle what Mark spam
does and put it on the message bar. It is a review dialog over the case
directory: show the bodies, allow editing, show what could not be resolved,
and call `submit` on approval, then show per-destination results.

It offers no retry button. A `deferred` destination belongs to cron, and a
button beside it would race the scheduled run and risk a duplicate report; the
dialog says when the retry is due instead. A `failed` or `in-flight`
destination needs the user to act somewhere else, not to press a button that
would repeat what already did not work. Sized M in the backlog, not the S a plain button would be:
a review dialog is real UI, and the user asked for the review to happen in the
application rather than in an editor.

The wire contract is this document's manifest plus the `abusectl` command name
in qtmaildir's config, in the shape `[sync] command` and the per-account
`send_command` already use.

## What each later spec has to settle

- **`contacts`**: RDAP bootstrap and referral chasing, caching policy, rate
  limits, and what happens for a netblock that publishes no abuse contact.
- **`report`**: the X-ARF (RFC 5965) schema version, which fields the user's
  reporting identity fills, and the plain-text alternative for desks that do
  not parse X-ARF.
- **`submit` and `retry`**: per-vendor auth and payload shapes, which
  rate-limit headers each vendor actually sends and how a soft failure is
  told from a hard one, the backoff curve, MISP event structure
  (one event per case, or per campaign, and how the dedup query decides),
  and how mail to an abuse desk is sent, which most likely reuses qtmaildir's
  own `send_command` rather than adding an SMTP client.
- **The qtmaildir dialog**: its own item, after 187 and 190.