aboutsummaryrefslogtreecommitdiffstats
path: root/docs/specs
diff options
context:
space:
mode:
authorDanilo M. <danix@danix.xyz>2026-09-10 16:22:10 +0200
committerDanilo M. <danix@danix.xyz>2026-09-10 16:22:10 +0200
commiteebf1f3fcc794b8a1700fe27b49c1bac202f89b8 (patch)
tree7e0db634398e027c47a63769d1fa27633244f664 /docs/specs
parentf1d706040785c90f114f80e7f45c597af06a03a9 (diff)
downloadabusectl-eebf1f3fcc794b8a1700fe27b49c1bac202f89b8.tar.gz
abusectl-eebf1f3fcc794b8a1700fe27b49c1bac202f89b8.zip
docs: settle the vendor and MISP destinations
The report spec names three destination kinds and the module writes only one. Verified through the public API: generate() on a manifest with an IP contact returns a single email row, and nothing in the tree mentions a misp or api row. The plan the module was built from never carried that task, so report is complete against its plan and incomplete against its spec. Found when the submit brainstorm began, since submit is specified to fill bodies into rows that do not exist. The addendum settles which rows exist: configured AND holding an indicator the destination accepts, each row carrying only the types it can act on. The accepted types were read from each vendor's own documentation rather than from memory, which is the provider table's lesson. That reading is also what removed VirusTotal: its only submission endpoints are POST /urls and file upload, so it accepts exactly what URLhaus accepts and nothing else, and it takes no verdict with a submission. It goes to backlog item 6 with the findings, including the reason its more interesting read side collides with the fourth property. Also records that generate() rebuilds destinations wholesale, which is safe only while the freeze rule holds, and names that as a constraint on the submit spec rather than a description of this one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0176FYdVfpzUq8S9jecqQqL6
Diffstat (limited to 'docs/specs')
-rw-r--r--docs/specs/2026-09-09-report.md280
1 files changed, 280 insertions, 0 deletions
diff --git a/docs/specs/2026-09-09-report.md b/docs/specs/2026-09-09-report.md
index b3a67ae..9861e64 100644
--- a/docs/specs/2026-09-09-report.md
+++ b/docs/specs/2026-09-09-report.md
@@ -448,3 +448,283 @@ Named here because this spec creates them, not to settle them:
`api` destination rows; whether it also learns to write their bodies is a
decision for that spec. If a vendor's payload turns out to need a value the
manifest does not hold, that is a change here, not a workaround there.
+
+---
+
+# Addendum, agreed 2026-09-10: the vendor and MISP destinations
+
+Status: **agreed 2026-09-10**, in one brainstorming session with the user.
+
+This addendum completes the spec above rather than superseding it. Everything
+before this line stands as written; what follows settles the part of the
+"Destinations" section that was specified and never built.
+
+## Why this exists
+
+The destination table above names three kinds, `email`, `api` and `misp`, and
+says the `api` and `misp` rows are "created as destination entries with their
+IOC lists and a `pending` status" so that "review can already see which vendors
+a case will reach".
+
+**`report` writes only `email` rows.** Verified through the public API on
+2026-09-10: `generate()` on a manifest with one IP contact returns
+`destinations` containing exactly one row, of kind `email`. Nothing in the
+module or the suite mentions a `misp` or `api` row; the only occurrences of
+`misp` in the tree are `init.py`'s config carry-across.
+
+This was not a defect in what was built. The implementation plan
+(`docs/plans/2026-09-09-report.md`) carried thirteen tasks and none of them was
+this one, so the module is complete against its plan and incomplete against its
+spec. The gap was found when the `submit` brainstorm began, because `submit`
+is specified to fill bodies into rows that do not exist.
+
+The consequence while it stands is that review is less honest than the spec
+promised: the user approving a case sees the abuse desks it will mail and no
+indication that the same case is bound for MISP and two vendors.
+
+## Which destinations, and what each accepts
+
+Three destinations. VirusTotal was considered and is deferred; see backlog
+item 6 for the findings, which are worth reading before it is reconsidered.
+
+The accepted types below were read from each vendor's own API documentation on
+2026-09-10, not written from memory. That distinction is the provider table's
+lesson (`init.PROVIDERS`): the first draft of that table was written from
+memory and every single range was wrong.
+
+| destination | accepts | auth header | endpoint | source |
+|---|---|---|---|---|
+| `misp` | every IOC type | `Authorization` | `POST /events`, `/attributes/add/` | MISP book, circl.lu/doc/misp/automation |
+| `abusedb` | `ipv4`, `ipv6` | `Key` | `POST /api/v2/report` | docs.abuseipdb.com |
+| `urlhaus` | `url` | `Auth-Key` | `POST https://urlhaus.abuse.ch/api/` | abuse.ch's own `submit_url.py` |
+
+The endpoints and headers are recorded here because they were checked in the
+same pass, not because `report` uses them. **`report` uses only the "accepts"
+column.** Resolving an endpoint is `submit`'s, and the table is here so that
+spec starts from a checked source rather than re-deriving one.
+
+**URLhaus's full submission documentation is behind a login at
+`auth.abuse.ch`.** The row above comes from the reference `submit_url.py` that
+abuse.ch publishes, which is a primary source but a narrower one than the docs.
+Re-verify the payload against the real documentation when the user has an
+account; that is a `submit` task, since only the accepted TYPE matters here and
+that much is unambiguous.
+
+## The rule: configured AND acceptable
+
+A destination earns a row when **both** hold:
+
+1. it is configured, meaning its config section is present and carries every
+ key that section requires (`api_key` for the two vendors, both `url` and
+ `api_key` for MISP; a section present with only one half of the MISP pair
+ is an error rather than a skip, see Config below), and
+2. the case holds at least one IOC of a type it accepts.
+
+Either alone is the wrong rule. A row for an unconfigured vendor is a promise
+that can only fail. A row for a configured vendor with nothing it can take is
+a promise with no content: an AbuseIPDB submission for a case whose only
+indicators are URLs has nothing to put in its `ip` parameter.
+
+**Each row carries only the IOCs its own destination accepts**, not the case's
+whole list. A row is what `submit` iterates, and a row listing a URL under
+AbuseIPDB is a submission that will be built wrong or dropped at send time,
+whichever the implementer notices first. MISP's row therefore carries every
+IOC and the vendor rows carry subsets, which is also what makes the ordering
+rule legible during review: MISP holds the whole incident, the vendors hold
+the parts they can act on.
+
+## The row shape
+
+```json
+{ "id": "misp", "kind": "misp", "iocs": ["ioc-1","ioc-2","ioc-3"],
+ "body": null, "status": "pending" },
+{ "id": "abusedb", "kind": "api", "iocs": ["ioc-1"],
+ "body": null, "status": "pending" },
+{ "id": "urlhaus", "kind": "api", "iocs": ["ioc-3"],
+ "body": null, "status": "pending" }
+```
+
+The same keys as an email row, so `submit` stays one loop with one ordering
+rule, as the umbrella design requires.
+
+**The id is the destination's name, a fixed literal, not a derived hash.** An
+email destination hashes its target address because a desk is discovered per
+case and its address is the only stable thing about it. There is exactly one
+AbuseIPDB. A literal id is also what lets `retry` find a row across cases
+without knowing how it was built.
+
+**There is no `target`.** An email row carries the abuse address it will mail.
+A vendor's endpoint is a property of the vendor rather than of the case, and
+writing it into the manifest would put a submission URL inside the evidence
+record, where a hand-edit or a corrupted file could redirect a submission to
+somewhere the user never named. `submit` resolves the endpoint from the
+destination module.
+
+**`body` stays null and there is no `body_sha256`.** The spec above defers the
+vendor payload shapes to `submit`, and the hash exists to record what was
+disclosed. Neither has anything to record yet. Writing a vendor's JSON now
+would mean guessing an endpoint's contract, which is the mistake this document
+already refuses once.
+
+## Rebuilding, and what makes it safe
+
+**`generate()` rebuilds `destinations[]` wholesale on every run**, email rows
+and vendor rows alike. There is no status-preserving merge and no matching by
+id.
+
+Today the module already assigns over the key, which was found during this
+session to discard a hand-constructed row marked `sent`. That is not a live
+defect: nothing has ever sent, because `submit` does not exist, so no manifest
+anywhere carries a status other than `pending` and there is no data to
+preserve.
+
+**This is safe only while the freeze rule holds**, and that dependency is
+stated rather than left implicit. Once a destination has landed, the case is
+frozen and `report` refuses to run at all, so a rebuild can never reach a
+manifest holding a receipt. The freeze marker is written by `submit`, which is
+unbuilt, so this is a constraint on that spec and not merely a description of
+this one: **if `submit` ships without writing `frozen` atomically with its
+first `sent` status, this rebuild becomes a data-loss bug.** It is named again
+in the handover section below.
+
+A status-preserving merge was considered and rejected. It would be a second
+mechanism enforcing what the freeze already enforces, and two mechanisms for
+one rule is how a derived check quietly comes to disagree with the explicit
+one. The spec above rejected a derived freeze check for the same reason.
+
+## Nothing configured
+
+`report` **warns and continues.** It writes a line to stderr naming the config
+file and saying no reporting destinations are configured, then exits zero.
+
+Not an error, on the spec's own rule that `report` "exits non-zero only when it
+could not write". A case with abuse-desk destinations and no vendors is a
+perfectly good report, and the tool is specified to be useful with no API key
+configured anywhere: `parse` + `contacts` + `report` produces a document that
+can be sent by hand.
+
+Not silent either, because the failure it hides is the one worth catching. A
+user who believes MISP is configured and finds no `misp` row has a typo'd
+section name, and the umbrella design makes MISP the gate for everything
+irreversible. A case built with no gate, discovered at submit time, is
+discovered a step too late.
+
+## Config
+
+`config.py` grows one more thing it reads: which destinations are configured.
+
+```toml
+[misp]
+url = "https://misp.example.invalid"
+api_key = "..."
+
+[abusedb]
+api_key = "..."
+
+[urlhaus]
+api_key = "..."
+```
+
+MISP takes a `url` because it is self-hosted and has no default endpoint. The
+two vendors take a key only; their endpoints are fixed and belong to their
+destination modules.
+
+The rules are the ones the file already follows, applied unchanged. A skipped
+answer is ABSENT, never an empty string. A non-string value is REJECTED with a
+message naming the file, not silently dropped, on the same reasoning that
+governs `[reporter]`: a value that reads as plausible and is wrong is worse
+than an error, and `ipaddress.ip_network(42)` is the trap this file learned it
+from. A section that is not a table is rejected the way `reporter = "A
+Reporter"` already is.
+
+**MISP's two keys are validated as a PAIR, and WHERE that is reported
+matters.** A url with no key, or a key with no url, is configured-and-broken
+rather than not-configured. This is the one place the absent-means-skipped
+convention needs help: absence of the whole section is a skip, absence of one
+half of it is a mistake.
+
+It must NOT be raised from `config.load()`. Every subcommand loads the config,
+so a `ValueError` there would make `parse` and `contacts` refuse to run over a
+half-filled MISP section that neither of them reads, and `cli.py` catches only
+`NotConfigured`, so the user would get a traceback rather than a sentence. The
+same trap the file already records for `[reporter]`, one level up: an answer
+reported somewhere other than where it is used.
+
+So `config.load()` reads the section and reports what it found; the
+half-filled pair is refused by `report`, which is the command that acts on it,
+with the not-configured exit code and a sentence naming the file and the
+missing key. `init` refuses it earlier still, at the prompt that asked.
+
+**Everything else inside those tables is carried through untouched.** `submit`
+will want timeouts, category codes and whatever else each API turns out to
+need, and this addendum deliberately does not design that. `config.py` reads
+what decides whether a row exists and leaves the rest alone, exactly as it
+leaves other sections alone today.
+
+## `init` prompts
+
+`init` grows four questions: the MISP url and key, the AbuseIPDB key, the
+URLhaus key. All four are skippable, and they say so, because the relay
+question and the hop picker now say they are required and a user reads the
+difference.
+
+The MISP pair is validated together at the point it is asked, so answering a
+url and skipping the key re-asks rather than writing a broken section. That is
+this repository's standing prompt rule, learned from a hand test that found
+four instances of one mistake: an answer validated somewhere other than where
+it was given costs the whole run.
+
+The config carry-across already preserves sections `build()` does not produce,
+and `init.py`'s own tests cover a `[misp]` key surviving an init that only sets
+the relays. Those tests describe the behaviour this addendum now makes real
+rather than hypothetical.
+
+**These prompts are HAND-TESTED by the user, not unit-tested**, like every
+other prompt in the tool. Whether a question reads clearly has no assertion.
+
+**A key in the config is a secret at rest.** The file is already mode 0600 and
+so are its backups, which is why that was built before any key existed. No
+change, named here so it is not mistaken for an oversight.
+
+## Testing
+
+Tested, because each has a right answer:
+
+- a configured destination with an acceptable IOC gets a row; one without an
+ acceptable IOC does not, and the URL-less case getting no `urlhaus` row is
+ the one to write first, since it is the rule's whole point
+- an unconfigured destination gets no row, however many IOCs would suit it
+- each row carries only the types its destination accepts, asserted against a
+ case holding an IP, a domain and a URL at once, which is the only shape that
+ can distinguish all three
+- MISP's row carries every IOC, including the domain neither vendor takes
+- a second `generate()` on the same manifest and config produces the same rows
+- the no-destination warning is written and the exit code is still zero
+- the config rejects a non-string key and a non-table section, each naming the
+ file
+- a half-filled MISP pair is refused by `report` with the not-configured exit
+ code, and does NOT stop `parse` from running, which is the half that would
+ regress silently
+
+Not tested: the `init` prompts, per the rule above.
+
+**No sweep is required for this change.** Both sweeps exist to prove that no
+recipient identifier reaches an IOC or a report body, and this addendum
+touches neither `parse.py` nor the content of any body. It adds rows to a
+manifest from values the user typed into their own config. Stated explicitly
+because `AGENTS.md` requires a sweep when `parse.py` changes, and a reader
+should be able to see that the question was asked rather than skipped.
+
+## What this leaves for `submit`
+
+Unchanged from the section above, plus one constraint this addendum creates:
+
+- **the `frozen` marker must be written in the same atomic manifest write as
+ the first `sent` status.** The spec above already required this; this
+ addendum now depends on it, because `generate()` rebuilds `destinations[]`
+ wholesale and the freeze is the only thing standing between that rebuild and
+ a manifest holding receipts.
+- the vendor payload shapes, per destination, against each vendor's real
+ documentation. URLhaus's needs an `auth.abuse.ch` account to read properly.
+- whether `report` learns to write those bodies, or `submit` does. The rows
+ now exist either way, which was the point of building them here.