path: root/docs/specs/2026-09-09-contacts.md

# abusectl `contacts`: IOCs to abuse contacts, over RDAP

Status: **agreed 2026-09-09**, in one brainstorming session with the user.
This spec settles the part the umbrella design left open, named there as
"RDAP bootstrap and referral chasing, caching policy, rate limits, and what
happens for a netblock that publishes no abuse contact".

Read `docs/specs/2026-09-08-abusectl-design.md` first. This document assumes
its manifest format, its case directory and its ordering between the parts,
and changes none of them.

## What it does

```
abusectl contacts <case>
```

Reads a case manifest, resolves an abuse contact for every IP and domain
indicator via RDAP, and rewrites the manifest with a `contacts[]` array. It
is read-only against the network: it queries registries and sends no mail,
files no report and changes nothing a third party can observe beyond the
queries themselves.

It is the FIRST network module in this tool. Everything below that looks
defensive is defensive because of that.

## Modules

```
abusectl/
  contacts.py     IOCs -> contacts[]            network, read-only
  rdap.py         bootstrap, query, jCard       network, read-only
```

`rdap.py` is the PROTOCOL: fetching and caching the IANA bootstrap files,
selecting a server for an address or a TLD, issuing a query, and pulling an
abuse address out of a jCard. `contacts.py` is the POLICY: which indicators
are resolvable, how hosts fold together, what is written into the manifest.

**The split is the same one that puts `redact.py` beside `parse.py`.** The
protocol dirt is where the defects will be, and a module of its own gets
tests that name it rather than tests that reach it incidentally through case
handling.

### The offline guarantee survives, for the whole suite

Both modules take a `fetch` callable as an ARGUMENT, defaulting to the real
urllib transport. Tests pass a fake and never construct the real one.

The suite must keep passing with `socket.socket`, `socket.create_connection`
and `socket.getaddrinfo` all raising, exactly as it does today, and that
property stays a property of the WHOLE suite. Not "the whole suite except
contacts". A network module that can only be tested with a network is a
module that stops being tested.

This is the pattern the umbrella design promised for exactly this moment: if
a later part needs the network, it goes in its own module with an injected
transport.

## THE FOURTH PROPERTY

The umbrella design states three properties that are not negotiable. This
part adds a fourth, because it is the first part that can violate it.

### Never disclose more than the host under investigation

An RDAP query tells a third party what the user is looking at. Queries carry
BARE HOSTS AND IP ADDRESSES ONLY, never a full URL.

A URL path can carry recipient identity. `parse.suspect_path_segments()`
FLAGS those rather than redacting them, deliberately, because a path segment
may be the thing being reported. That decision is safe only while the URL
stays local. Sending a URL to a registry would leak precisely what property 1
exists to prevent, through a channel property 1 does not cover: property 1
governs what is PUBLISHED, and a query is a disclosure that never appears in
any report.

Concretely: a `url` indicator contributes its HOST to the domain worklist and
nothing else. Path, query and fragment never leave the machine.

**This is a trap, not a theoretical concern.** The obvious implementation
resolves "a contact for each indicator" by reading each indicator's `value`,
and for a `url` indicator that value is an entire URL. The sweep described
under Testing exists to catch exactly that mistake against real mail.

## What gets resolved

| Indicator | Query | Why |
|---|---|---|
| `ipv4`, `ipv6` | RDAP IP lookup | the sending IP is the primary takedown target |
| `domain` | RDAP domain lookup | the registrar holds the abuse desk |
| `url` | its HOST, as a domain | folded into the domain worklist, see the fourth property |
| `sha256` | none | no registry owns a hash |
| `observation` | none | not a network object |

Indicators carrying `confidence: untrusted-hop` ARE resolved. A forged chain's
IP may still be the real sender's, and refusing to resolve it would discard
the case the tool exists for. The confidence marker stays in the manifest, so
review shows the user that a contact came from an attacker-supplied hop
before any report goes out.

**Hosts fold.** Twenty URLs on one host produce one worklist entry and one
query. The resulting contact entry lists every indicator id that contributed,
so nothing is lost by folding.

## RDAP mechanics

### Bootstrap

Fetch `ipv4.json`, `ipv6.json` and `dns.json` from `data.iana.org/rdap/`.
Cache them under `$XDG_CACHE_HOME/abusectl/rdap/`, defaulting to
`~/.cache/abusectl/rdap/`. TTL 7 days.

The cache is NOT in the case directory. It is not evidence, it is a copy of a
public map.

**A failed refetch falls back to the stale copy**, with a warning, rather
than failing the run. Last week's map is almost certainly still correct, and
IANA being unreachable should not stop the user filing a report. A stale
bootstrap is safe to serve: the worst case is querying a server that has
moved, which fails and reads as "no contact".

### Server selection

- IP: longest-prefix match over the bootstrap ranges, using `ipaddress`.
- Domain: exact TLD match in `dns.json`.
- No match: `abuse: []` with the reason recorded. Many TLDs publish no RDAP
  service at all, and that is a normal outcome rather than a defect.

### The registrable domain, by walking up

RDAP domain lookup wants the registrable domain. `mail.deep.example.invalid`
is not registrable and most registries answer it with a 404.

Query the full host, then drop one label and retry, until a server answers or
the labels run out. Cap at 5 attempts and NEVER query a bare TLD.

**This replaces a Public Suffix List deliberately.** Taking the last two
labels is wrong for every multi-part suffix, `co.uk` and `com.au` among them,
and a bundled PSL snapshot is a transcribed table that goes stale weekly,
which is the failure `init.PROVIDERS` already documents at length. The
registry is the authority on what is registrable, so the walk asks it instead
of approximating it.

The in-memory dedup below makes the walk cheap in the common case: a
campaign's twenty subdomains under one registrable domain collapse to one
successful query plus a few cheap misses.

### Transport

`urllib.request` from the standard library. The tool still declares no
dependencies, and there is no `requirements.txt` in the repository to add one
to.

- `Accept: application/rdap+json, application/json;q=0.9`. The weaker
  alternative is offered because some registries serve plain
  `application/json`, and a strict header would lose a contact that was
  published.
- **timeout mandatory**, 10 seconds. `urllib` with no timeout blocks forever,
  and a hung registry would hang a review.
- **redirects capped at 5 hops.** RIRs redirect to each other and the default
  handler follows without a limit of ours.
- **an https to http downgrade is refused.**

The cap and the downgrade refusal follow the discipline `_MAX_REDIRECT_DEPTH`
already sets in `parse.py`, and for the same reason: this is remote data
directing our next move.

`requests` was considered and rejected. It is present in slackware64-current,
so packaging was not the deciding factor. It simply buys little here: the two
things this module needs beyond a plain GET are the redirect cap and the
downgrade refusal, and neither is a knob on `requests` either, both needing a
`Session` plus custom wiring of comparable size. Declaring no dependencies at
all means the tool runs and tests with no venv step.

### Caching policy

Bootstrap: on disk, 7 day TTL, as above.

Responses: **in memory only, keyed by query URL, for the duration of one
run.** Nothing on disk.

The asymmetry is deliberate and is about which staleness hurts. A stale
bootstrap fails loudly enough, producing a miss. **A stale abuse address
sends a report into a mailbox nobody reads, silently**, and the user learns
nothing came of it only by never hearing back. RDAP rate limits are generous
for a human-paced tool filing a handful of reports, so the saving is not
worth that.

In-run dedup is free and still worth having: it is what stops a case with
forty URLs on one host from issuing forty identical queries.

## Extracting the address

Strict role match, and nothing else. An entity whose `roles` contains `abuse`
yields the `email` entries of its jCard. Recurse into nested `entities`,
depth capped at 4, because the abuse entity is usually a child of the
organisation entity.

Four rules, each with a victim:

**1. No fallback to a non-abuse entity.** A `technical` or `registrant`
contact is a named human who never volunteered to receive abuse mail. Mailing
them is useless and is a small privacy harm to an uninvolved third party.

**2. No `abuse@<domain>` by convention.** RFC 2142 says the mailbox should
exist; for a phishing domain it belongs to the ATTACKER. Constructing it
would mail the attacker to tell them their campaign was caught and that the
user's address is live, which is the exact outcome property 2 exists to
prevent. This is the single worst thing this module could do, and it looks
like a helpful fallback, which is why it is written down.

**3. Never follow a `links` referral to fetch a contact.** Some responses
point elsewhere for an entity. Following that is an outbound fetch to a URL
the response chose for us, which is SSRF-shaped. If the address is not in the
response, there is no address.

**4. Validate the address before storing it.** `email.utils.parseaddr`,
exactly one address, and reject any control character or newline. It becomes
a mail recipient in `report` and `submit`; a CRLF there is header injection
into mail this tool sends.

**Multiple abuse addresses are all kept.** Some netblocks publish two, and
picking one arbitrarily can drop the desk that would have answered.

**A missing contact is a normal outcome, not a failure.** The umbrella design
already settles this: the indicator keeps an empty `abuse` with the reason,
no email destination is created for it, and review shows what could not be
resolved. The indicator still reaches MISP and the vendor feeds.

## The manifest

```json
"contacts": [
  { "iocs": ["ioc-1"], "query": "198.51.100.7",
    "abuse": ["abuse@example.invalid"],
    "source": "rdap", "handle": "AS64496",
    "server": "rdap.example.invalid" },
  { "iocs": ["ioc-3", "ioc-7"], "query": "example.invalid",
    "abuse": [], "source": "rdap",
    "error": "no abuse role published" }
]
```

Two departures from the sketch in the umbrella design, both deliberate:

- **`iocs` is a list**, because hosts fold and one contact can serve several
  indicators.
- **`abuse` is a list**, because multiple desks are real.

`query` records what was actually asked. Review can then see that a URL
indicator was resolved by its host, which is the fourth property made visible
rather than merely promised.

`case.py` remains the only writer, and the write stays atomic.

### Re-running

**A re-run overwrites `contacts[]` wholesale**, in a single atomic write
after every lookup has finished.

Never a merge. A merge lets a contact resolved a week ago survive into a
report filed today, which is the stale-address hazard the response caching
policy already refuses. Overwriting makes a re-run always safe and always
current, which matters because a partial network failure makes re-running the
natural next step.

### Failure is per query

A timeout or an error on one indicator records that reason against that entry
and the run continues. The manifest is written once at the end regardless. A
run that resolved nothing at all still writes, so that review shows why
rather than showing an absent section.

## Testing

TDD, as everywhere in this repository. Write the failing test, watch it fail,
implement, watch it pass.

`rdap.py` is tested with a fake `fetch` returning canned responses in real RIR
SHAPES, with every address, handle and range replaced by `example.invalid`
and RFC 5737 documentation values, per the fixture rule in AGENTS.md.

Tests that have a right answer:

- longest-prefix server selection, including an address matching two ranges
- the label walk stops at the first server that answers, and never queries a
  bare TLD
- nested entity recursion finds an abuse entity under an organisation
- a response carrying only a `technical` role yields no address, not the
  technical one
- a `links` referral is NOT fetched: the fake transport asserts it was never
  called for that URL
- a CRLF in an email field is rejected
- **a `url` indicator contributes only its host**: the fake transport asserts
  no query ever contained the path
- a stale bootstrap is used when the refetch fails
- the whole suite still passes with sockets raising

### Two sweeps of the user's real spam

The `parse` sweep found a header the parser never read and a spoofed
`Reply-To` no fixture had. The same corpus is worth sweeping here, but this
module sends traffic, so it splits in two.

**Sweep A, offline, the whole corpus.** Build the worklist from all messages
with a fake transport that RECORDS queries and returns canned responses. No
packets. This carries the assertion:

> no query string ever contained anything but a bare host or IP address

That is the fourth property checked against real mail rather than against
fixtures, over the shapes a synthetic fixture misses: a URL with a path, with
userinfo, an IDN, a trailing dot, a bracketed IPv6 literal, a host that is
itself an address. It also yields free counts: unique hosts, how many
indicators fold per host, how deep the label walks go.

**Sweep B, online, a deliberate handful.** Ten to twenty indicators picked by
hand across distinct netblocks and TLDs, against the real registries. This
confirms that real jCard shapes parse, that the walk terminates, and that the
selection works. Twenty queries is polite and unremarkable.

Sweep A carries the assertion. Sweep B is a shape check.

**Ask before either, and sweep B needs an explicit go-ahead**, because unlike
every test before it in this repository it is OUTWARD-FACING: it discloses to
registries, and to a registrar who may be the attacker's, which netblocks and
domains the user is investigating, from their address, at a known time. That
cannot be undone, which is why it stays small and hand-picked rather than
being run over the corpus.

What may leave the sweep scripts: counts, tallies, TLDs, error reasons, walk
depths, and whether any query was ever malformed. What may not: an address, a
real abuse contact, a real domain written into a test, or a URL from a real
message. A defect found by a sweep is reproduced as a synthetic fixture and
THAT is committed. The real messages stay in the scratchpad, which is
per-session and outside the repository.

## Deliberately absent

- **No response cache on disk.** See the caching policy above.
- **No PSL.** See the label walk above.
- **No retry or backoff here.** Rate-limit handling belongs to `submit` and
  `retry`, which own the deadline logic the umbrella design specifies. A
  read-only lookup that fails is simply re-run.
- **No `whois` fallback.** It is a different protocol with an unparseable
  free-text response per registry, and its absence costs a contact rather
  than a report.
- **No contact for `sha256` or `observation` indicators.** No registry owns
  them.