diff options
| -rw-r--r-- | AGENTS.md | 94 | ||||
| -rw-r--r-- | README.md | 66 | ||||
| -rw-r--r-- | abusectl/rdap.py | 14 | ||||
| -rw-r--r-- | docs/BACKLOG.md | 29 |
4 files changed, 170 insertions, 33 deletions
@@ -37,7 +37,7 @@ python3 -m unittest tests.test_parse -v Run the tool from the checkout with `python3 -m abusectl`. -## THREE PROPERTIES THAT ARE NOT NEGOTIABLE +## FOUR PROPERTIES THAT ARE NOT NEGOTIABLE Each has a concrete victim. Do not weaken one for convenience, and do not "simplify" the code that enforces it without reading this section first. @@ -112,6 +112,47 @@ With no boundary configured, `parse` REFUSES rather than guessing the outermost public IP. Refusing is only defensible because `abusectl init` is the route out, which is why that command exists at all. +### 4. A query carries a bare host or IP, never a URL + +`contacts` is the first part that talks to anyone. An RDAP query discloses to a +third party which host or netblock the user is investigating, so it carries a +BARE HOST OR IP ADDRESS and nothing else. + +**Property 1 does not cover this.** Property 1 governs what is PUBLISHED, and a +query appears in no report at all. A URL path can carry recipient identity, and +`suspect_path_segments` deliberately FLAGS a suspect segment rather than +redacting it, which is safe only while the URL stays local. The moment a URL +becomes a query it stops being local, and the flag protects nobody. The victim +is the recipient whose address sits in a path segment and is handed to a +registrar, and, when the registrar is the attacker's own, the reporter, who has +just told the attacker they are under investigation. + +`contacts.is_queryable()` is the SINGLE ADMISSION POINT. Every branch of +`worklist()` funnels through `add()`, which refuses a value that is not a bare +host or IP and keeps it as `unusable` so the user sees it during review. Do not +move that check into a branch, and do not add a branch that reaches `rdap` +without passing through it. + +**That rule exists because the property leaked three times, each time the same +shape.** First the `url` branch was guarded with a host-only extraction and the +`domain` branch was not, so `From: Bank <phish@victim%40example.org.invalid>` +sent the recipient's own address to a registry. Then the `domain` branch was +guarded at one admission point and the IP branch still leaked, because +`_is_ip()` sat INSIDE that point and returned True before the validator ran, and +`ipaddress.ip_address()` accepts an IPv6 scope id whose content is unconstrained +free text: `fe80::1%victim@example.org` parses as valid. The lesson is both +halves of that: **validation applied per branch gets forgotten on the next +branch, and a check that short-circuits before the validator is the same bug +wearing a different coat.** + +`rdap._quoted()` percent-encodes a component so it cannot escape its path +segment. That contains URL-STRUCTURE attacks, traversal and an injected query or +fragment; it does NOT reduce disclosure, because an encoded address is still the +address. It is not a second line of defence for this property, and the third +leak came from treating it as one: `safe=""` encoded the colons RFC 9082 +requires be literal, so every IPv6 query silently 404ed and read as "no abuse +desk published". + ## Architecture ``` @@ -122,10 +163,12 @@ abusectl/ parse.py .eml -> IOCs pure, offline redact.py the safety rule, alone and testable case.py case directory: create, manifest read/write, atomic + contacts.py IOCs -> abuse contacts network, read-only + rdap.py bootstrap, query, jCard network, read-only ``` -Planned, each needing its own spec first: `contacts` (RDAP), `report` (X-ARF), -`submit` (MISP then vendors), `retry` (cron). See the design document. +Planned, each needing its own spec first: `report` (X-ARF), `submit` (MISP then +vendors), `retry` (cron). See the design document. **`redact.py` is separate from `parse.py` deliberately.** It is the safety property, and a module of its own gets tests that name it rather than tests @@ -265,6 +308,36 @@ When a sweep finds a defect, reproduce it as a synthetic fixture under the rule above and commit THAT. The real message stays in the scratchpad, which is per-session and outside the repository. +**A module that talks to third parties gets TWO sweeps, and they are not the +same shape.** `contacts` was verified this way and any later network part +should be too. + +Sweep A is OFFLINE and covers the whole corpus. It builds the worklist for +every message and asserts the fourth property against it, with no query issued. +Breadth is free because nothing leaves the machine. What it observed: 92 +messages, 1685 indicators, 0 crashes, 0 empty parses, 110 unique query targets, +1206 indicators folded away by host deduplication, and 0 malformed queries. The +second assertion took all 548 addresses appearing in the raw sources and checked +each against every query string: no address reached a query, and neither did any +local part of six characters or more. + +Sweep B is ONLINE, hand-picked and deliberately SMALL, because every query in it +discloses to a third party and cannot be undone. Twelve targets, chosen to +exercise the branches rather than to cover volume: 12 of 12 completed, 0 +failures. All five RIRs, RIPE, ARIN, APNIC, LACNIC and AFRINIC, returned a +parseable jCard. IPv6 resolved live. The label walk reduced `www.ripe.net` and +`a.b.c.example.org` to two labels, and `nic.uk` resolved, which is the +multi-part suffix case. Three non-resolutions were correct behaviour rather than +failures: AFRINIC and `nic.cz` publish a handle but no `abuse` role, and +`example.museum` has no RDAP server for the TLD. The IANA bootstrap held 5 IPv4 +services, 5 IPv6 and 590 DNS. + +**Sweep B must never draw its targets from the user's own spam corpus.** A query +tells a registrar which of their customers someone is investigating, and for a +phishing domain that registrar may be the attacker's own. Pick targets that are +public infrastructure or documentation names, and keep the corpus to sweep A, +where nothing is sent. + ## Working on this repo Work directly on `master`, no PR flow. Commits are GPG-signed (`git commit -S`); @@ -273,9 +346,9 @@ rejection is correct until proven otherwise. `HANDOFF.md` is local-only and gitignored; never stage or commit it. -Design first for anything unbuilt: `contacts`, `report`, `submit` and `retry` -each need their own spec before code, because each has real unknowns. The -umbrella design settles only what they share. +Design first for anything unbuilt: `report`, `submit` and `retry` each need +their own spec before code, because each has real unknowns. The umbrella design +settles only what they share. ## Documents @@ -283,11 +356,14 @@ umbrella design settles only what they share. before changing anything about the case format, the redaction rule, or the ordering between MISP and the vendors. - `docs/specs/2026-09-09-contacts.md`, the `contacts` spec. Read it before - touching RDAP, the bootstrap cache, or anything that issues a query: it adds - a FOURTH non-negotiable property, that a query carries a bare host or IP and - never a URL. + touching RDAP, the bootstrap cache, or anything that issues a query. The + fourth property it introduced is stated above in its own right; the spec + carries the reasoning behind the rest of the module. - `docs/plans/2026-09-08-parse.md`, the plan `init` and `parse` were built from. Historical once built, but it records why each test exists. +- `docs/plans/2026-09-09-contacts.md`, the plan `contacts` and `rdap` were + built from. Historical in the same way, and it records the three leaks the + fourth property above is written from. - `docs/BACKLOG.md`, open items, with the cause verified in the code rather than assumed. Read it before starting work; add to it rather than fixing something unasked. @@ -4,9 +4,9 @@ Abuse reporting for phishing mail. Parses a flagged message, extracts its indicators, resolves who to report each one to, and files the result to a MISP instance and to public abuse channels. -**Status: `init` and `parse` are built.** The rest of the pipeline is designed -but not written, see `docs/specs/2026-09-08-abusectl-design.md`. Nothing here -submits anything to anyone yet. +**Status: `init`, `parse` and `contacts` are built.** The rest of the pipeline +is designed but not written, see `docs/specs/2026-09-08-abusectl-design.md`. +Nothing here submits anything to anyone yet. ## What it does @@ -20,10 +20,10 @@ abusectl submit <case> # MISP, then the vendors network, writes abusectl retry # whatever is due cron ``` -`init` and `parse` exist today. Each subcommand runs on its own and is useful -on its own: `parse` triages a message with no keys configured at all, and -`parse` plus `contacts` plus `report` will produce a document you can send by -hand before any API key exists. +`init`, `parse` and `contacts` exist today. Each subcommand runs on its own and +is useful on its own: `parse` triages a message with no keys configured at all, +and `parse` plus `contacts` plus `report` will produce a document you can send +by hand before any API key exists. State lives in a **case directory** rather than in memory, so a review can take a week and survive a reboot. @@ -61,6 +61,34 @@ it. Either way the old file is copied to `config.toml.bak-<timestamp>` first, and any section this run does not set, such as `[misp]`, is carried across untouched. +## Resolving the contacts + +```bash +abusectl contacts ~/.local/share/abusectl/2026-09-08-a3f1 +``` + +`contacts` reads the indicators `parse` recorded and asks the registries who to +report each one to, over RDAP. This is the first command that needs **network +access**, and no API key: RDAP is public. + +It is **read-only**. It queries registries and sends nothing to anyone, so +running it does not report you, or the message, to anybody. A query carries a +bare host or IP address and never a URL, so the path and query string of a +phishing link stay on your machine. Twenty URLs on one host produce one query. + +**A re-run is safe.** It overwrites the contacts in the manifest rather than +merging into them, so a run interrupted halfway leaves nothing half-merged, and +running it again after a registry recovers simply replaces the result. The +indicators themselves are untouched. + +**A missing abuse contact is a normal outcome, not an error.** Many TLDs publish +no RDAP service at all, and a registry that answers may publish a handle with no +`abuse` role on it. Those are recorded with the reason, and the run continues: +one indicator failing must not cost the contacts that did resolve. An indicator +that is not a bare host or IP is kept as unusable, with why, rather than +silently dropped, so you can see during review whether the parser mangled a real +host or the sender planted something. + ## What a case looks like ``` @@ -83,7 +111,7 @@ Every indicator says where it came from. An IP from the trust boundary is `boundary-hop`, the one address that can be stood behind; anything below it is `untrusted-hop`, recorded because it may be useful but never presented as fact. -## Two properties that are not negotiable +## Three properties that are not negotiable **Recipient identifiers are never captured.** Not the `To`, `Cc`, `Delivered-To` or `X-Original-To` headers, not your Message-IDs, not maildir @@ -98,6 +126,12 @@ redirect chains, not remote images. Following a link confirms your address is live to the sender and fires exactly the tracker the message wanted. Redirect chains are read from headers and link text, never by following them. +**A registry query carries a bare host or IP, never a URL.** `contacts` is the +first command that talks to anyone, and an RDAP query tells a third party what +you are looking at. A URL path can carry your identity, and the registrar you +ask may be the attacker's own, so the path, query and fragment of a phishing +link never leave your machine. Only the host does. + ## Reversible and irreversible `submit` writes to MISP first and stops if that fails. MISP is your own @@ -115,10 +149,11 @@ gets its own spec before it is built. ## Requirements -Python 3.12 or newer, and nothing else for what is built today: `init` and -`parse` are standard library only. `contacts` will need an HTTP client and -`submit` will need PyMISP, so a venv in the checkout is the development -arrangement; packaging comes once the tool does something worth installing. +Python 3.12 or newer, and nothing else for what is built today: `init`, `parse` +and `contacts` are standard library only, `contacts` included, which reaches the +registries with `urllib`. `submit` will need PyMISP, so a venv in the checkout is +the development arrangement; packaging comes once the tool does something worth +installing. ## Tests @@ -126,12 +161,13 @@ arrangement; packaging comes once the tool does something worth installing. python3 -m unittest discover tests ``` -100 tests, no framework, no network. Two of them are not ordinary unit tests +192 tests, no framework, no network. Two of them are not ordinary unit tests and are the ones worth knowing about. The `Received`-chain test is mutation-checked: walking one hop too far makes it report an innocent party named in a header the attacker wrote, and the test fails if that regresses. -And the parser's suite passes with `socket` disabled entirely, so -"nothing is fetched" is verified rather than documented. +And the suite passes with `socket` disabled entirely, `contacts` and its RDAP +code included, so "nothing is fetched" is verified rather than documented. The +network goes in through a `fetch` argument the tests replace. ## License diff --git a/abusectl/rdap.py b/abusectl/rdap.py index 286d960..02045b4 100644 --- a/abusectl/rdap.py +++ b/abusectl/rdap.py @@ -29,7 +29,12 @@ redirect is remote data directing our next request. This follows the discipline _MAX_REDIRECT_DEPTH already sets in parse.py. """ +import email.utils +import ipaddress import json +import os +import pathlib +import time import urllib.error import urllib.parse import urllib.request @@ -73,10 +78,6 @@ def http_fetch(url: str) -> dict: return json.loads(response.read()) -import os -import pathlib -import time - _BOOTSTRAP_TTL = 7 * 24 * 60 * 60 _BOOTSTRAP_URL = "https://data.iana.org/rdap/{}.json" _REGISTRIES = ("ipv4", "ipv6", "dns") @@ -132,9 +133,6 @@ def bootstrap(registry: str, cache_root=None, fetch=http_fetch) -> dict: return data -import ipaddress - - def _service_entries(bootstrap_data): """Yield (keys, urls) for every well-formed entry, skipping the rest. @@ -235,8 +233,6 @@ def server_for_tld(tld: str, bootstrap_data: dict) -> str | None: return None -import email.utils - _MAX_ENTITY_DEPTH = 4 diff --git a/docs/BACKLOG.md b/docs/BACKLOG.md index 25793e1..79c0ab1 100644 --- a/docs/BACKLOG.md +++ b/docs/BACKLOG.md @@ -6,6 +6,7 @@ number and gains a status rather than being renumbered. | # | Item | Size | Status | |---|------|------|--------| | 1 | Skip boilerplate namespace URLs | XS | open | +| 2 | An IDN indicator resolves to no contact | S | open | ## 1. Skip boilerplate namespace URLs @@ -38,3 +39,31 @@ skip-list is a thing an attacker knows will not be reported. If it is built, the skipped URLs should still be visible somewhere during review rather than silently dropped, on the same reasoning that makes `suspect_path_segments` flag rather than redact. + +## 2. An IDN indicator resolves to no contact + +**Observed.** `contacts.is_queryable()` refuses any non-ASCII value, so a +phishing domain in an internationalised name is recorded as `unusable` with the +reason "not ASCII, and we do not guess at an IDN encoding", and no registry is +asked. The `ponytail:` comment on that function names this as the ceiling it +accepted. + +**Cause.** Deliberate, not a defect. Converting a name to punycode means +choosing an encoding for attacker-supplied text, and a wrong choice sends a +different name than the one in the message to a registry, which is a disclosure +made about the wrong party. Refusing keeps the value in front of the user +instead. + +**Approach.** `value.encode("idna")` is the obvious move and is not enough on +its own: it normalises, so the name queried may differ from the name written in +the message, and the manifest must record BOTH, the way `query_domain` already +records `queried` separately when the label walk shortens a host. The refusal +reason already distinguishes non-ASCII from illegal, so the user-facing half +exists. + +**Constraints.** Not a leak: the failure direction is asking nobody, which is +safe. Weigh against a real argument for leaving it: a homograph name is exactly +where the attacker wants the tool to normalise on their behalf, and a +consultant chasing one indicator by hand is a smaller cost than a query made +about a name the user never saw. Wait for a real IDN indicator in a sweep before +building it. |