diff options
Diffstat (limited to 'docs/specs')
| -rw-r--r-- | docs/specs/2026-09-09-report.md | 280 |
1 files changed, 280 insertions, 0 deletions
diff --git a/docs/specs/2026-09-09-report.md b/docs/specs/2026-09-09-report.md index b3a67ae..9861e64 100644 --- a/docs/specs/2026-09-09-report.md +++ b/docs/specs/2026-09-09-report.md @@ -448,3 +448,283 @@ Named here because this spec creates them, not to settle them: `api` destination rows; whether it also learns to write their bodies is a decision for that spec. If a vendor's payload turns out to need a value the manifest does not hold, that is a change here, not a workaround there. + +--- + +# Addendum, agreed 2026-09-10: the vendor and MISP destinations + +Status: **agreed 2026-09-10**, in one brainstorming session with the user. + +This addendum completes the spec above rather than superseding it. Everything +before this line stands as written; what follows settles the part of the +"Destinations" section that was specified and never built. + +## Why this exists + +The destination table above names three kinds, `email`, `api` and `misp`, and +says the `api` and `misp` rows are "created as destination entries with their +IOC lists and a `pending` status" so that "review can already see which vendors +a case will reach". + +**`report` writes only `email` rows.** Verified through the public API on +2026-09-10: `generate()` on a manifest with one IP contact returns +`destinations` containing exactly one row, of kind `email`. Nothing in the +module or the suite mentions a `misp` or `api` row; the only occurrences of +`misp` in the tree are `init.py`'s config carry-across. + +This was not a defect in what was built. The implementation plan +(`docs/plans/2026-09-09-report.md`) carried thirteen tasks and none of them was +this one, so the module is complete against its plan and incomplete against its +spec. The gap was found when the `submit` brainstorm began, because `submit` +is specified to fill bodies into rows that do not exist. + +The consequence while it stands is that review is less honest than the spec +promised: the user approving a case sees the abuse desks it will mail and no +indication that the same case is bound for MISP and two vendors. + +## Which destinations, and what each accepts + +Three destinations. VirusTotal was considered and is deferred; see backlog +item 6 for the findings, which are worth reading before it is reconsidered. + +The accepted types below were read from each vendor's own API documentation on +2026-09-10, not written from memory. That distinction is the provider table's +lesson (`init.PROVIDERS`): the first draft of that table was written from +memory and every single range was wrong. + +| destination | accepts | auth header | endpoint | source | +|---|---|---|---|---| +| `misp` | every IOC type | `Authorization` | `POST /events`, `/attributes/add/` | MISP book, circl.lu/doc/misp/automation | +| `abusedb` | `ipv4`, `ipv6` | `Key` | `POST /api/v2/report` | docs.abuseipdb.com | +| `urlhaus` | `url` | `Auth-Key` | `POST https://urlhaus.abuse.ch/api/` | abuse.ch's own `submit_url.py` | + +The endpoints and headers are recorded here because they were checked in the +same pass, not because `report` uses them. **`report` uses only the "accepts" +column.** Resolving an endpoint is `submit`'s, and the table is here so that +spec starts from a checked source rather than re-deriving one. + +**URLhaus's full submission documentation is behind a login at +`auth.abuse.ch`.** The row above comes from the reference `submit_url.py` that +abuse.ch publishes, which is a primary source but a narrower one than the docs. +Re-verify the payload against the real documentation when the user has an +account; that is a `submit` task, since only the accepted TYPE matters here and +that much is unambiguous. + +## The rule: configured AND acceptable + +A destination earns a row when **both** hold: + +1. it is configured, meaning its config section is present and carries every + key that section requires (`api_key` for the two vendors, both `url` and + `api_key` for MISP; a section present with only one half of the MISP pair + is an error rather than a skip, see Config below), and +2. the case holds at least one IOC of a type it accepts. + +Either alone is the wrong rule. A row for an unconfigured vendor is a promise +that can only fail. A row for a configured vendor with nothing it can take is +a promise with no content: an AbuseIPDB submission for a case whose only +indicators are URLs has nothing to put in its `ip` parameter. + +**Each row carries only the IOCs its own destination accepts**, not the case's +whole list. A row is what `submit` iterates, and a row listing a URL under +AbuseIPDB is a submission that will be built wrong or dropped at send time, +whichever the implementer notices first. MISP's row therefore carries every +IOC and the vendor rows carry subsets, which is also what makes the ordering +rule legible during review: MISP holds the whole incident, the vendors hold +the parts they can act on. + +## The row shape + +```json +{ "id": "misp", "kind": "misp", "iocs": ["ioc-1","ioc-2","ioc-3"], + "body": null, "status": "pending" }, +{ "id": "abusedb", "kind": "api", "iocs": ["ioc-1"], + "body": null, "status": "pending" }, +{ "id": "urlhaus", "kind": "api", "iocs": ["ioc-3"], + "body": null, "status": "pending" } +``` + +The same keys as an email row, so `submit` stays one loop with one ordering +rule, as the umbrella design requires. + +**The id is the destination's name, a fixed literal, not a derived hash.** An +email destination hashes its target address because a desk is discovered per +case and its address is the only stable thing about it. There is exactly one +AbuseIPDB. A literal id is also what lets `retry` find a row across cases +without knowing how it was built. + +**There is no `target`.** An email row carries the abuse address it will mail. +A vendor's endpoint is a property of the vendor rather than of the case, and +writing it into the manifest would put a submission URL inside the evidence +record, where a hand-edit or a corrupted file could redirect a submission to +somewhere the user never named. `submit` resolves the endpoint from the +destination module. + +**`body` stays null and there is no `body_sha256`.** The spec above defers the +vendor payload shapes to `submit`, and the hash exists to record what was +disclosed. Neither has anything to record yet. Writing a vendor's JSON now +would mean guessing an endpoint's contract, which is the mistake this document +already refuses once. + +## Rebuilding, and what makes it safe + +**`generate()` rebuilds `destinations[]` wholesale on every run**, email rows +and vendor rows alike. There is no status-preserving merge and no matching by +id. + +Today the module already assigns over the key, which was found during this +session to discard a hand-constructed row marked `sent`. That is not a live +defect: nothing has ever sent, because `submit` does not exist, so no manifest +anywhere carries a status other than `pending` and there is no data to +preserve. + +**This is safe only while the freeze rule holds**, and that dependency is +stated rather than left implicit. Once a destination has landed, the case is +frozen and `report` refuses to run at all, so a rebuild can never reach a +manifest holding a receipt. The freeze marker is written by `submit`, which is +unbuilt, so this is a constraint on that spec and not merely a description of +this one: **if `submit` ships without writing `frozen` atomically with its +first `sent` status, this rebuild becomes a data-loss bug.** It is named again +in the handover section below. + +A status-preserving merge was considered and rejected. It would be a second +mechanism enforcing what the freeze already enforces, and two mechanisms for +one rule is how a derived check quietly comes to disagree with the explicit +one. The spec above rejected a derived freeze check for the same reason. + +## Nothing configured + +`report` **warns and continues.** It writes a line to stderr naming the config +file and saying no reporting destinations are configured, then exits zero. + +Not an error, on the spec's own rule that `report` "exits non-zero only when it +could not write". A case with abuse-desk destinations and no vendors is a +perfectly good report, and the tool is specified to be useful with no API key +configured anywhere: `parse` + `contacts` + `report` produces a document that +can be sent by hand. + +Not silent either, because the failure it hides is the one worth catching. A +user who believes MISP is configured and finds no `misp` row has a typo'd +section name, and the umbrella design makes MISP the gate for everything +irreversible. A case built with no gate, discovered at submit time, is +discovered a step too late. + +## Config + +`config.py` grows one more thing it reads: which destinations are configured. + +```toml +[misp] +url = "https://misp.example.invalid" +api_key = "..." + +[abusedb] +api_key = "..." + +[urlhaus] +api_key = "..." +``` + +MISP takes a `url` because it is self-hosted and has no default endpoint. The +two vendors take a key only; their endpoints are fixed and belong to their +destination modules. + +The rules are the ones the file already follows, applied unchanged. A skipped +answer is ABSENT, never an empty string. A non-string value is REJECTED with a +message naming the file, not silently dropped, on the same reasoning that +governs `[reporter]`: a value that reads as plausible and is wrong is worse +than an error, and `ipaddress.ip_network(42)` is the trap this file learned it +from. A section that is not a table is rejected the way `reporter = "A +Reporter"` already is. + +**MISP's two keys are validated as a PAIR, and WHERE that is reported +matters.** A url with no key, or a key with no url, is configured-and-broken +rather than not-configured. This is the one place the absent-means-skipped +convention needs help: absence of the whole section is a skip, absence of one +half of it is a mistake. + +It must NOT be raised from `config.load()`. Every subcommand loads the config, +so a `ValueError` there would make `parse` and `contacts` refuse to run over a +half-filled MISP section that neither of them reads, and `cli.py` catches only +`NotConfigured`, so the user would get a traceback rather than a sentence. The +same trap the file already records for `[reporter]`, one level up: an answer +reported somewhere other than where it is used. + +So `config.load()` reads the section and reports what it found; the +half-filled pair is refused by `report`, which is the command that acts on it, +with the not-configured exit code and a sentence naming the file and the +missing key. `init` refuses it earlier still, at the prompt that asked. + +**Everything else inside those tables is carried through untouched.** `submit` +will want timeouts, category codes and whatever else each API turns out to +need, and this addendum deliberately does not design that. `config.py` reads +what decides whether a row exists and leaves the rest alone, exactly as it +leaves other sections alone today. + +## `init` prompts + +`init` grows four questions: the MISP url and key, the AbuseIPDB key, the +URLhaus key. All four are skippable, and they say so, because the relay +question and the hop picker now say they are required and a user reads the +difference. + +The MISP pair is validated together at the point it is asked, so answering a +url and skipping the key re-asks rather than writing a broken section. That is +this repository's standing prompt rule, learned from a hand test that found +four instances of one mistake: an answer validated somewhere other than where +it was given costs the whole run. + +The config carry-across already preserves sections `build()` does not produce, +and `init.py`'s own tests cover a `[misp]` key surviving an init that only sets +the relays. Those tests describe the behaviour this addendum now makes real +rather than hypothetical. + +**These prompts are HAND-TESTED by the user, not unit-tested**, like every +other prompt in the tool. Whether a question reads clearly has no assertion. + +**A key in the config is a secret at rest.** The file is already mode 0600 and +so are its backups, which is why that was built before any key existed. No +change, named here so it is not mistaken for an oversight. + +## Testing + +Tested, because each has a right answer: + +- a configured destination with an acceptable IOC gets a row; one without an + acceptable IOC does not, and the URL-less case getting no `urlhaus` row is + the one to write first, since it is the rule's whole point +- an unconfigured destination gets no row, however many IOCs would suit it +- each row carries only the types its destination accepts, asserted against a + case holding an IP, a domain and a URL at once, which is the only shape that + can distinguish all three +- MISP's row carries every IOC, including the domain neither vendor takes +- a second `generate()` on the same manifest and config produces the same rows +- the no-destination warning is written and the exit code is still zero +- the config rejects a non-string key and a non-table section, each naming the + file +- a half-filled MISP pair is refused by `report` with the not-configured exit + code, and does NOT stop `parse` from running, which is the half that would + regress silently + +Not tested: the `init` prompts, per the rule above. + +**No sweep is required for this change.** Both sweeps exist to prove that no +recipient identifier reaches an IOC or a report body, and this addendum +touches neither `parse.py` nor the content of any body. It adds rows to a +manifest from values the user typed into their own config. Stated explicitly +because `AGENTS.md` requires a sweep when `parse.py` changes, and a reader +should be able to see that the question was asked rather than skipped. + +## What this leaves for `submit` + +Unchanged from the section above, plus one constraint this addendum creates: + +- **the `frozen` marker must be written in the same atomic manifest write as + the first `sent` status.** The spec above already required this; this + addendum now depends on it, because `generate()` rebuilds `destinations[]` + wholesale and the freeze is the only thing standing between that rebuild and + a manifest holding receipts. +- the vendor payload shapes, per destination, against each vendor's real + documentation. URLhaus's needs an `auth.abuse.ch` account to read properly. +- whether `report` learns to write those bodies, or `submit` does. The rows + now exist either way, which was the point of building them here. |
