diff options
| -rw-r--r-- | AGENTS.md | 251 | ||||
| -rw-r--r-- | CLAUDE.md | 8 |
2 files changed, 259 insertions, 0 deletions
diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..a5f195c --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,251 @@ +# AGENTS.md + +This file is the single source of truth for how to work on this repository, +shared across every agent tool (Claude Code, opencode, etc). `CLAUDE.md` beside +it is a pointer to this file and holds no content of its own. + +## What this is + +`abusectl` parses a phishing email, extracts indicators of compromise, resolves +who to report each one to, and files the result to a MISP instance and to +public abuse channels. It exists because the author is a security consultant +who has been phished and wants to act quickly on a campaign that targets them. + +It is a **sidecar to qtmaildir**, not part of it. qtmaildir does no network +protocol work at all by design, and this needs RDAP, three vendor APIs and mail +to abuse desks. qtmaildir invokes this tool by name, the way it invokes +`mailsync.sh`, and hosts the review dialog. The two repositories are coupled +only by the manifest format and a command name in qtmaildir's config; there is +no submodule and there should not be one. + +Backlog item 194 in qtmaildir's +`docs/superpowers/plans/2026-08-03-post-0.1.0-usability.md` carries the +qtmaildir half. + +## Build and test + +```bash +python3 -m unittest discover tests +``` + +Stdlib only, no framework, no network, no fixtures directory beyond +`tests/fixtures/*.eml`. A single module: + +```bash +python3 -m unittest tests.test_parse -v +``` + +Run the tool from the checkout with `python3 -m abusectl`. + +## THREE PROPERTIES THAT ARE NOT NEGOTIABLE + +Each has a concrete victim. Do not weaken one for convenience, and do not +"simplify" the code that enforces it without reading this section first. + +### 1. Recipient identifiers must never reach a report + +Reports go to third parties and to public feeds. A phishing URL routinely +carries the recipient's identity in its query string (`?e=<address>`, +`?u=<base64>`, `?id=<md5>`), so publishing a URL verbatim leaks the victim's +address, and it **deanonymises the reporter to the attacker**, since abuse +desks forward reports to the abused customer and URLhaus is public. + +The guarantee is STRUCTURAL: the tool cannot disclose an identifier it was +never given. `parse.py` does not read `To`, `Cc`, `Delivered-To` or +`X-Original-To` at all, and `redact.py` blanks values before anything is +stored. + +`redact.url()` keeps parameter NAMES (they fingerprint the kit) and blanks +VALUES (they identify the recipient). It covers the query, the fragment AND +userinfo. The fragment matters even though a browser never sends it, because +what gets published is the URL TEXT from the message. Userinfo is worse than +the rest: it carries an address and a credential together. + +A token is unique per recipient by design, so keeping it would make +correlation WORSE, not better: two messages from one campaign would look like +different URLs. + +**The one exception is the rule's own logic, not a hole in it.** A redirector +carries its destination in a parameter. That destination is an indicator rather +than a recipient identifier, so `redact.url_valued_parameters()` recovers it +and it becomes an IOC in its own right, still redacted itself. Every value that +is not a URL stays blanked. + +Path segments are FLAGGED, never redacted (`suspect_path_segments`), because +unlike a query value a path segment may be the thing being reported. Flagged is +visible to the user during review; silent was the bug. + +### 2. Nothing is ever fetched or resolved + +Not URLs, not redirect chains, not remote images, no DNS. Following a link +confirms to the sender that the address is live and fires exactly the tracker +the message wanted. + +A redirect chain is read from what the message DECLARES. `redirect_chains()` +recurses into recovered targets, bounded by `_MAX_REDIRECT_DEPTH` and a `seen` +set, because a nested value is attacker-supplied. + +**This is verified, not asserted.** The suite passes with `socket.socket`, +`socket.create_connection` and `socket.getaddrinfo` all raising. Keep that +possible: if a later part needs the network, it goes in its own module with an +injected transport, never in the parse path. + +### 3. The trust boundary is configured, never guessed + +`Received` headers are prepended, so the list runs newest first: our own +infrastructure at the top, the sender below. **Everything below our own servers +was written by whoever was talking to them and can be forged wholesale.** +Attackers routinely prepend headers naming innocent third parties. + +`sending_ip()` returns the FIRST hop outside the configured networks, walking +outermost inward. Walking to the last one instead reports whoever the attacker +chose. `tests/fixtures/forged-chain.eml` exists for exactly this: the correct +answer is `203.0.113.99`, and `198.51.100.7` is a planted innocent party. + +**That test is mutation-checked and must stay that way.** Change `sending_ip()` +to keep walking and `test_a_forged_chain_stops_at_the_first_untrusted_hop` must +fail with `'198.51.100.8' != '203.0.113.99'`. A test that cannot fail is not +protecting anything, and this is the one standing between the tool and +reporting an innocent party. + +With no boundary configured, `parse` REFUSES rather than guessing the outermost +public IP. Refusing is only defensible because `abusectl init` is the route out, +which is why that command exists at all. + +## Architecture + +``` +abusectl/ + cli.py argparse dispatch, exit codes, prompts. No logic. + init.py first-run config: pure builder + prompt helpers + provider table + config.py reads ~/.config/abusectl/config.toml via tomllib + parse.py .eml -> IOCs pure, offline + redact.py the safety rule, alone and testable + case.py case directory: create, manifest read/write, atomic +``` + +Planned, each needing its own spec first: `contacts` (RDAP), `report` (X-ARF), +`submit` (MISP then vendors), `retry` (cron). See the design document. + +**`redact.py` is separate from `parse.py` deliberately.** It is the safety +property, and a module of its own gets tests that name it rather than tests +that reach it incidentally. + +**`parse.py` takes the trust boundary as an ARGUMENT, never a config read.** +That is what keeps it pure and testable with no files on disk. `cli.py` reads +the config and passes it in. + +## The case directory + +``` +<cases>/2026-09-08-a3f1/ + source.eml the original, UNREDACTED + manifest.json IOCs, and later contacts and per-destination status + bodies/ report bodies, once `report` exists +``` + +`case.py` is the ONLY writer. The manifest is written atomically, temp file +plus rename, because a half-written manifest during a review is a corrupted +evidence record. A failed write must leave the existing manifest untouched and +no stray temp file; there is a test for both. + +**Nothing deletes a case.** No cleanup function exists and none should be +added: these are the user's working evidence. + +**`source.eml` is unredacted**, so a case directory is sensitive at rest. The +redaction rules are about what may be PUBLISHED. The submit path must never +attach `source.eml` wholesale. + +`load()` refuses a manifest whose `format` it does not know, rather than +silently dropping fields a newer writer added. + +## Config + +`~/.config/abusectl/config.toml`, read with stdlib `tomllib`, written by +`init.py`. Mode `0600`, and so are the backups: the file holds API keys as +later parts land. + +**A skipped answer is ABSENT from the file, never an empty string.** +`api_key = ""` reads as configured-and-broken and produces a confusing auth +error much later; absent reads as not-configured and the part that wants it +can say so plainly. `config.load()` enforces the same rule reading back: an +empty `cases` falls back to the default rather than becoming `Path("")`, which +would scatter evidence into whatever directory the command ran from. + +Two traps found by hand-testing and now covered: + +- `trusted_relays = "192.0.2.0/24"` without brackets is valid TOML and + iterates CHARACTERS, validating `'1'`, `'9'`, `'2'`. Rejected explicitly now. +- **`ipaddress.ip_network(42)` does not raise**, it returns `0.0.0.42/32`. A + typo'd config would have produced a valid-looking boundary trusting an + address the user never named. Non-string entries are rejected. + +`init` never overwrites silently: it refuses without `--force`, the interactive +path shows what is configured and asks, and either route backs the old file up +first. **Sections `build()` does not produce are carried across verbatim**, so +an init that only sets the relays cannot discard a `[misp]` key set earlier. + +## The provider table + +`init.PROVIDERS` holds each provider's published sending ranges, transcribed +from their own SPF records on 2026-09-08. The transcription commands are in a +comment above it. + +**Do not edit these from memory.** The first draft was written from memory and +every single range was wrong: the table claimed eleven IPv4 ranges for Gmail +where `_spf.google.com` publishes two. A wrong range means a hop is treated as +the user's own and the real sender is never reported. + +Google's `goog.json` looks authoritative and is the WRONG source: it lists all +Google infrastructure, over a hundred ranges including `8.8.8.8`, and using it +would trust every Google-hosted service as part of the user's mail path. +`_spf.google.com` is the mail-sending answer. + +IPv4 only, deliberately. An IPv6 hop from a listed provider simply does not +match and the user is asked instead, which is the safe direction. + +## Testing + +TDD. Write the failing test, watch it fail, implement, watch it pass. + +**Test what has a right answer.** The geometry of a manifest, the generated +TOML, the redaction of a URL, the hop a chain resolves to. Do not write a test +that passes regardless of what the code does. + +**The interactive prompts are HAND-TESTED by the user, not unit-tested.** +Whether a question reads clearly has no assertion, and a test driving stdin +asserts the wording it was written against and breaks on a rewording that +improved it. `cli.py`'s prompt helpers are covered by the user running them; +its dispatch is covered by `tests/test_cli.py`. + +That hand test found five defects in one pass, four of them the same mistake: +an answer validated somewhere other than where it was given, so a typo cost the +whole run. **Validate each answer at the prompt that asked for it** and re-ask, +rather than erroring after the next question. + +**Fixtures use `example.org`, `.invalid` and RFC 5737 documentation ranges +only.** A real phishing sample carries the identifiers this tool exists to keep +out of reports, and a repository is potentially public. Check a fixture's +weekday with `date -d <yyyy-mm-dd> +%A` rather than writing it from memory: an +RFC2822 parser validates the day against the date and a wrong one reads as a +malformed header. + +## Working on this repo + +Work directly on `master`, no PR flow. Commits are GPG-signed (`git commit -S`); +never pass `--no-verify`. Global git hooks scan for personal data and a +rejection is correct until proven otherwise. + +`HANDOFF.md` is local-only and gitignored; never stage or commit it. + +Design first for anything unbuilt: `contacts`, `report`, `submit` and `retry` +each need their own spec before code, because each has real unknowns. The +umbrella design settles only what they share. + +## Documents + +- `docs/specs/2026-09-08-abusectl-design.md` — the umbrella design. Read it + before changing anything about the case format, the redaction rule, or the + ordering between MISP and the vendors. +- `docs/plans/2026-09-08-parse.md` — the plan `init` and `parse` were built + from. Historical once built, but it records why each test exists. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..7a26e74 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,8 @@ +# CLAUDE.md + +This file is intentionally thin. AGENTS.md is the single source of truth for +how to work on this repository, shared across every agent tool (Claude Code, +opencode, etc). Do not re-populate this file with duplicated content, edit +AGENTS.md instead. + +@AGENTS.md |