path: root/AGENTS.md

# AGENTS.md

This file is the single source of truth for how to work on this repository,
shared across every agent tool (Claude Code, opencode, etc). `CLAUDE.md` beside
it is a pointer to this file and holds no content of its own.

## What this is

`abusectl` parses a phishing email, extracts indicators of compromise, resolves
who to report each one to, and files the result to a MISP instance and to
public abuse channels. It exists because the author is a security consultant
who has been phished and wants to act quickly on a campaign that targets them.

It is a **sidecar to qtmaildir**, not part of it. qtmaildir does no network
protocol work at all by design, and this needs RDAP, three vendor APIs and mail
to abuse desks. qtmaildir invokes this tool by name, the way it invokes
`mailsync.sh`, and hosts the review dialog. The two repositories are coupled
only by the manifest format and a command name in qtmaildir's config; there is
no submodule and there should not be one.

Backlog item 194 in qtmaildir's
`docs/superpowers/plans/2026-08-03-post-0.1.0-usability.md` carries the
qtmaildir half.

## Build and test

```bash
python3 -m unittest discover tests
```

Stdlib only, no framework, no network, no fixtures directory beyond
`tests/fixtures/*.eml`. A single module:

```bash
python3 -m unittest tests.test_parse -v
```

Run the tool from the checkout with `python3 -m abusectl`.

## THREE PROPERTIES THAT ARE NOT NEGOTIABLE

Each has a concrete victim. Do not weaken one for convenience, and do not
"simplify" the code that enforces it without reading this section first.

### 1. Recipient identifiers must never reach a report

Reports go to third parties and to public feeds. A phishing URL routinely
carries the recipient's identity in its query string (`?e=<address>`,
`?u=<base64>`, `?id=<md5>`), so publishing a URL verbatim leaks the victim's
address, and it **deanonymises the reporter to the attacker**, since abuse
desks forward reports to the abused customer and URLhaus is public.

The guarantee is STRUCTURAL: the tool cannot disclose an identifier it was
never given. `parse.py` does not read `To`, `Cc`, `Delivered-To` or
`X-Original-To` at all, and `redact.py` blanks values before anything is
stored.

`redact.url()` keeps parameter NAMES (they fingerprint the kit) and blanks
VALUES (they identify the recipient). It covers the query, the fragment AND
userinfo. The fragment matters even though a browser never sends it, because
what gets published is the URL TEXT from the message. Userinfo is worse than
the rest: it carries an address and a credential together.

A token is unique per recipient by design, so keeping it would make
correlation WORSE, not better: two messages from one campaign would look like
different URLs.

**The one exception is the rule's own logic, not a hole in it.** A redirector
carries its destination in a parameter. That destination is an indicator rather
than a recipient identifier, so `redact.url_valued_parameters()` recovers it
and it becomes an IOC in its own right, still redacted itself. Every value that
is not a URL stays blanked.

Path segments are FLAGGED, never redacted (`suspect_path_segments`), because
unlike a query value a path segment may be the thing being reported. Flagged is
visible to the user during review; silent was the bug.

### 2. Nothing is ever fetched or resolved

Not URLs, not redirect chains, not remote images, no DNS. Following a link
confirms to the sender that the address is live and fires exactly the tracker
the message wanted.

A redirect chain is read from what the message DECLARES. `redirect_chains()`
recurses into recovered targets, bounded by `_MAX_REDIRECT_DEPTH` and a `seen`
set, because a nested value is attacker-supplied.

**This is verified, not asserted.** The suite passes with `socket.socket`,
`socket.create_connection` and `socket.getaddrinfo` all raising. Keep that
possible: if a later part needs the network, it goes in its own module with an
injected transport, never in the parse path.

### 3. The trust boundary is configured, never guessed

`Received` headers are prepended, so the list runs newest first: our own
infrastructure at the top, the sender below. **Everything below our own servers
was written by whoever was talking to them and can be forged wholesale.**
Attackers routinely prepend headers naming innocent third parties.

`sending_ip()` returns the FIRST hop outside the configured networks, walking
outermost inward. Walking to the last one instead reports whoever the attacker
chose. `tests/fixtures/forged-chain.eml` exists for exactly this: the correct
answer is `203.0.113.99`, and `198.51.100.7` is a planted innocent party.

**That test is mutation-checked and must stay that way.** Change `sending_ip()`
to keep walking and `test_a_forged_chain_stops_at_the_first_untrusted_hop` must
fail with `'198.51.100.8' != '203.0.113.99'`. A test that cannot fail is not
protecting anything, and this is the one standing between the tool and
reporting an innocent party.

With no boundary configured, `parse` REFUSES rather than guessing the outermost
public IP. Refusing is only defensible because `abusectl init` is the route out,
which is why that command exists at all.

## Architecture

```
abusectl/
  cli.py        argparse dispatch, exit codes, prompts. No logic.
  init.py       first-run config: pure builder + prompt helpers + provider table
  config.py     reads ~/.config/abusectl/config.toml via tomllib
  parse.py      .eml -> IOCs                       pure, offline
  redact.py     the safety rule, alone and testable
  case.py       case directory: create, manifest read/write, atomic
```

Planned, each needing its own spec first: `contacts` (RDAP), `report` (X-ARF),
`submit` (MISP then vendors), `retry` (cron). See the design document.

**`redact.py` is separate from `parse.py` deliberately.** It is the safety
property, and a module of its own gets tests that name it rather than tests
that reach it incidentally.

**`parse.py` takes the trust boundary as an ARGUMENT, never a config read.**
That is what keeps it pure and testable with no files on disk. `cli.py` reads
the config and passes it in.

## The case directory

```
<cases>/2026-09-08-a3f1/
  source.eml          the original, UNREDACTED
  manifest.json       IOCs, and later contacts and per-destination status
  bodies/             report bodies, once `report` exists
```

`case.py` is the ONLY writer. The manifest is written atomically, temp file
plus rename, because a half-written manifest during a review is a corrupted
evidence record. A failed write must leave the existing manifest untouched and
no stray temp file; there is a test for both.

**Nothing deletes a case.** No cleanup function exists and none should be
added: these are the user's working evidence.

**`source.eml` is unredacted**, so a case directory is sensitive at rest. The
redaction rules are about what may be PUBLISHED. The submit path must never
attach `source.eml` wholesale.

`load()` refuses a manifest whose `format` it does not know, rather than
silently dropping fields a newer writer added.

## Config

`~/.config/abusectl/config.toml`, read with stdlib `tomllib`, written by
`init.py`. Mode `0600`, and so are the backups: the file holds API keys as
later parts land.

**A skipped answer is ABSENT from the file, never an empty string.**
`api_key = ""` reads as configured-and-broken and produces a confusing auth
error much later; absent reads as not-configured and the part that wants it
can say so plainly. `config.load()` enforces the same rule reading back: an
empty `cases` falls back to the default rather than becoming `Path("")`, which
would scatter evidence into whatever directory the command ran from.

Two traps found by hand-testing and now covered:

- `trusted_relays = "192.0.2.0/24"` without brackets is valid TOML and
  iterates CHARACTERS, validating `'1'`, `'9'`, `'2'`. Rejected explicitly now.
- **`ipaddress.ip_network(42)` does not raise**, it returns `0.0.0.42/32`. A
  typo'd config would have produced a valid-looking boundary trusting an
  address the user never named. Non-string entries are rejected.

`init` never overwrites silently: it refuses without `--force`, the interactive
path shows what is configured and asks, and either route backs the old file up
first. **Sections `build()` does not produce are carried across verbatim**, so
an init that only sets the relays cannot discard a `[misp]` key set earlier.

## The provider table

`init.PROVIDERS` holds each provider's published sending ranges, transcribed
from their own SPF records on 2026-09-08. The transcription commands are in a
comment above it.

**Do not edit these from memory.** The first draft was written from memory and
every single range was wrong: the table claimed eleven IPv4 ranges for Gmail
where `_spf.google.com` publishes two. A wrong range means a hop is treated as
the user's own and the real sender is never reported.

Google's `goog.json` looks authoritative and is the WRONG source: it lists all
Google infrastructure, over a hundred ranges including `8.8.8.8`, and using it
would trust every Google-hosted service as part of the user's mail path.
`_spf.google.com` is the mail-sending answer.

IPv4 only, deliberately. An IPv6 hop from a listed provider simply does not
match and the user is asked instead, which is the safe direction.

## Testing

TDD. Write the failing test, watch it fail, implement, watch it pass.

**Test what has a right answer.** The geometry of a manifest, the generated
TOML, the redaction of a URL, the hop a chain resolves to. Do not write a test
that passes regardless of what the code does.

**The interactive prompts are HAND-TESTED by the user, not unit-tested.**
Whether a question reads clearly has no assertion, and a test driving stdin
asserts the wording it was written against and breaks on a rewording that
improved it. `cli.py`'s prompt helpers are covered by the user running them;
its dispatch is covered by `tests/test_cli.py`.

That hand test found five defects in one pass, four of them the same mistake:
an answer validated somewhere other than where it was given, so a typo cost the
whole run. **Validate each answer at the prompt that asked for it** and re-ask,
rather than erroring after the next question.

**Fixtures use `example.org`, `.invalid` and RFC 5737 documentation ranges
only.** A real phishing sample carries the identifiers this tool exists to keep
out of reports, and a repository is potentially public. Check a fixture's
weekday with `date -d <yyyy-mm-dd> +%A` rather than writing it from memory: an
RFC2822 parser validates the day against the date and a wrong one reads as a
malformed header.

**Sweep the user's real spam as a HAND TEST, and never as a fixture.** A
synthetic fixture only ever contains the shapes someone thought of, and the
shapes nobody thought of are the ones that leak. Real mail is the only source
of those, so the leak property is checked against `tag:spam` in the user's own
notmuch index. It is worth doing: the first message swept found a header the
parser did not read at all, and the corpus exercised a spoofed `Reply-To`
display name that no fixture had.

Ask before reading the user's mail. Then, from a script in the scratchpad and
never in the repo:

```python
raw = subprocess.run(["notmuch", "show", "--format=raw", mid],
                     capture_output=True, check=True).stdout
iocs = parse.iocs(raw, trusted=[...])
blob = repr(iocs)
for addr in set(re.findall(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+",
                           raw.decode("utf-8", "replace"))):
    assert addr not in blob, (mid, addr)
```

The assertion is the whole point and it must be that broad: compare EVERY
address in the raw source, headers and body, against the entire IOC output.
Checking only the recipient misses an address the parser invented from a
display name, which is exactly how the `_domain_of` defect reached a report.
Also count crashes and empty results; a message that parses to nothing is a
finding too.

What may leave that script: counts, header names, origin tallies, and a
domain with its local part removed. What may not: an address, a subject, a
Message-ID, a URL from a real message, or a real domain written into a test.
When a sweep finds a defect, reproduce it as a synthetic fixture under the
rule above and commit THAT. The real message stays in the scratchpad, which
is per-session and outside the repository.

## Working on this repo

Work directly on `master`, no PR flow. Commits are GPG-signed (`git commit -S`);
never pass `--no-verify`. Global git hooks scan for personal data and a
rejection is correct until proven otherwise.

`HANDOFF.md` is local-only and gitignored; never stage or commit it.

Design first for anything unbuilt: `contacts`, `report`, `submit` and `retry`
each need their own spec before code, because each has real unknowns. The
umbrella design settles only what they share.

## Documents

- `docs/specs/2026-09-08-abusectl-design.md`, the umbrella design. Read it
  before changing anything about the case format, the redaction rule, or the
  ordering between MISP and the vendors.
- `docs/specs/2026-09-09-contacts.md`, the `contacts` spec. Read it before
  touching RDAP, the bootstrap cache, or anything that issues a query: it adds
  a FOURTH non-negotiable property, that a query carries a bare host or IP and
  never a URL.
- `docs/plans/2026-09-08-parse.md`, the plan `init` and `parse` were built
  from. Historical once built, but it records why each test exists.
- `docs/BACKLOG.md`, open items, with the cause verified in the code rather
  than assumed. Read it before starting work; add to it rather than fixing
  something unasked.