aboutsummaryrefslogtreecommitdiffstats
path: root/docs/specs
diff options
context:
space:
mode:
Diffstat (limited to 'docs/specs')
-rw-r--r--docs/specs/2026-09-09-report.md280
1 files changed, 280 insertions, 0 deletions
diff --git a/docs/specs/2026-09-09-report.md b/docs/specs/2026-09-09-report.md
index b3a67ae..9861e64 100644
--- a/docs/specs/2026-09-09-report.md
+++ b/docs/specs/2026-09-09-report.md
@@ -448,3 +448,283 @@ Named here because this spec creates them, not to settle them:
`api` destination rows; whether it also learns to write their bodies is a
decision for that spec. If a vendor's payload turns out to need a value the
manifest does not hold, that is a change here, not a workaround there.
+
+---
+
+# Addendum, agreed 2026-09-10: the vendor and MISP destinations
+
+Status: **agreed 2026-09-10**, in one brainstorming session with the user.
+
+This addendum completes the spec above rather than superseding it. Everything
+before this line stands as written; what follows settles the part of the
+"Destinations" section that was specified and never built.
+
+## Why this exists
+
+The destination table above names three kinds, `email`, `api` and `misp`, and
+says the `api` and `misp` rows are "created as destination entries with their
+IOC lists and a `pending` status" so that "review can already see which vendors
+a case will reach".
+
+**`report` writes only `email` rows.** Verified through the public API on
+2026-09-10: `generate()` on a manifest with one IP contact returns
+`destinations` containing exactly one row, of kind `email`. Nothing in the
+module or the suite mentions a `misp` or `api` row; the only occurrences of
+`misp` in the tree are `init.py`'s config carry-across.
+
+This was not a defect in what was built. The implementation plan
+(`docs/plans/2026-09-09-report.md`) carried thirteen tasks and none of them was
+this one, so the module is complete against its plan and incomplete against its
+spec. The gap was found when the `submit` brainstorm began, because `submit`
+is specified to fill bodies into rows that do not exist.
+
+The consequence while it stands is that review is less honest than the spec
+promised: the user approving a case sees the abuse desks it will mail and no
+indication that the same case is bound for MISP and two vendors.
+
+## Which destinations, and what each accepts
+
+Three destinations. VirusTotal was considered and is deferred; see backlog
+item 6 for the findings, which are worth reading before it is reconsidered.
+
+The accepted types below were read from each vendor's own API documentation on
+2026-09-10, not written from memory. That distinction is the provider table's
+lesson (`init.PROVIDERS`): the first draft of that table was written from
+memory and every single range was wrong.
+
+| destination | accepts | auth header | endpoint | source |
+|---|---|---|---|---|
+| `misp` | every IOC type | `Authorization` | `POST /events`, `/attributes/add/` | MISP book, circl.lu/doc/misp/automation |
+| `abusedb` | `ipv4`, `ipv6` | `Key` | `POST /api/v2/report` | docs.abuseipdb.com |
+| `urlhaus` | `url` | `Auth-Key` | `POST https://urlhaus.abuse.ch/api/` | abuse.ch's own `submit_url.py` |
+
+The endpoints and headers are recorded here because they were checked in the
+same pass, not because `report` uses them. **`report` uses only the "accepts"
+column.** Resolving an endpoint is `submit`'s, and the table is here so that
+spec starts from a checked source rather than re-deriving one.
+
+**URLhaus's full submission documentation is behind a login at
+`auth.abuse.ch`.** The row above comes from the reference `submit_url.py` that
+abuse.ch publishes, which is a primary source but a narrower one than the docs.
+Re-verify the payload against the real documentation when the user has an
+account; that is a `submit` task, since only the accepted TYPE matters here and
+that much is unambiguous.
+
+## The rule: configured AND acceptable
+
+A destination earns a row when **both** hold:
+
+1. it is configured, meaning its config section is present and carries every
+ key that section requires (`api_key` for the two vendors, both `url` and
+ `api_key` for MISP; a section present with only one half of the MISP pair
+ is an error rather than a skip, see Config below), and
+2. the case holds at least one IOC of a type it accepts.
+
+Either alone is the wrong rule. A row for an unconfigured vendor is a promise
+that can only fail. A row for a configured vendor with nothing it can take is
+a promise with no content: an AbuseIPDB submission for a case whose only
+indicators are URLs has nothing to put in its `ip` parameter.
+
+**Each row carries only the IOCs its own destination accepts**, not the case's
+whole list. A row is what `submit` iterates, and a row listing a URL under
+AbuseIPDB is a submission that will be built wrong or dropped at send time,
+whichever the implementer notices first. MISP's row therefore carries every
+IOC and the vendor rows carry subsets, which is also what makes the ordering
+rule legible during review: MISP holds the whole incident, the vendors hold
+the parts they can act on.
+
+## The row shape
+
+```json
+{ "id": "misp", "kind": "misp", "iocs": ["ioc-1","ioc-2","ioc-3"],
+ "body": null, "status": "pending" },
+{ "id": "abusedb", "kind": "api", "iocs": ["ioc-1"],
+ "body": null, "status": "pending" },
+{ "id": "urlhaus", "kind": "api", "iocs": ["ioc-3"],
+ "body": null, "status": "pending" }
+```
+
+The same keys as an email row, so `submit` stays one loop with one ordering
+rule, as the umbrella design requires.
+
+**The id is the destination's name, a fixed literal, not a derived hash.** An
+email destination hashes its target address because a desk is discovered per
+case and its address is the only stable thing about it. There is exactly one
+AbuseIPDB. A literal id is also what lets `retry` find a row across cases
+without knowing how it was built.
+
+**There is no `target`.** An email row carries the abuse address it will mail.
+A vendor's endpoint is a property of the vendor rather than of the case, and
+writing it into the manifest would put a submission URL inside the evidence
+record, where a hand-edit or a corrupted file could redirect a submission to
+somewhere the user never named. `submit` resolves the endpoint from the
+destination module.
+
+**`body` stays null and there is no `body_sha256`.** The spec above defers the
+vendor payload shapes to `submit`, and the hash exists to record what was
+disclosed. Neither has anything to record yet. Writing a vendor's JSON now
+would mean guessing an endpoint's contract, which is the mistake this document
+already refuses once.
+
+## Rebuilding, and what makes it safe
+
+**`generate()` rebuilds `destinations[]` wholesale on every run**, email rows
+and vendor rows alike. There is no status-preserving merge and no matching by
+id.
+
+Today the module already assigns over the key, which was found during this
+session to discard a hand-constructed row marked `sent`. That is not a live
+defect: nothing has ever sent, because `submit` does not exist, so no manifest
+anywhere carries a status other than `pending` and there is no data to
+preserve.
+
+**This is safe only while the freeze rule holds**, and that dependency is
+stated rather than left implicit. Once a destination has landed, the case is
+frozen and `report` refuses to run at all, so a rebuild can never reach a
+manifest holding a receipt. The freeze marker is written by `submit`, which is
+unbuilt, so this is a constraint on that spec and not merely a description of
+this one: **if `submit` ships without writing `frozen` atomically with its
+first `sent` status, this rebuild becomes a data-loss bug.** It is named again
+in the handover section below.
+
+A status-preserving merge was considered and rejected. It would be a second
+mechanism enforcing what the freeze already enforces, and two mechanisms for
+one rule is how a derived check quietly comes to disagree with the explicit
+one. The spec above rejected a derived freeze check for the same reason.
+
+## Nothing configured
+
+`report` **warns and continues.** It writes a line to stderr naming the config
+file and saying no reporting destinations are configured, then exits zero.
+
+Not an error, on the spec's own rule that `report` "exits non-zero only when it
+could not write". A case with abuse-desk destinations and no vendors is a
+perfectly good report, and the tool is specified to be useful with no API key
+configured anywhere: `parse` + `contacts` + `report` produces a document that
+can be sent by hand.
+
+Not silent either, because the failure it hides is the one worth catching. A
+user who believes MISP is configured and finds no `misp` row has a typo'd
+section name, and the umbrella design makes MISP the gate for everything
+irreversible. A case built with no gate, discovered at submit time, is
+discovered a step too late.
+
+## Config
+
+`config.py` grows one more thing it reads: which destinations are configured.
+
+```toml
+[misp]
+url = "https://misp.example.invalid"
+api_key = "..."
+
+[abusedb]
+api_key = "..."
+
+[urlhaus]
+api_key = "..."
+```
+
+MISP takes a `url` because it is self-hosted and has no default endpoint. The
+two vendors take a key only; their endpoints are fixed and belong to their
+destination modules.
+
+The rules are the ones the file already follows, applied unchanged. A skipped
+answer is ABSENT, never an empty string. A non-string value is REJECTED with a
+message naming the file, not silently dropped, on the same reasoning that
+governs `[reporter]`: a value that reads as plausible and is wrong is worse
+than an error, and `ipaddress.ip_network(42)` is the trap this file learned it
+from. A section that is not a table is rejected the way `reporter = "A
+Reporter"` already is.
+
+**MISP's two keys are validated as a PAIR, and WHERE that is reported
+matters.** A url with no key, or a key with no url, is configured-and-broken
+rather than not-configured. This is the one place the absent-means-skipped
+convention needs help: absence of the whole section is a skip, absence of one
+half of it is a mistake.
+
+It must NOT be raised from `config.load()`. Every subcommand loads the config,
+so a `ValueError` there would make `parse` and `contacts` refuse to run over a
+half-filled MISP section that neither of them reads, and `cli.py` catches only
+`NotConfigured`, so the user would get a traceback rather than a sentence. The
+same trap the file already records for `[reporter]`, one level up: an answer
+reported somewhere other than where it is used.
+
+So `config.load()` reads the section and reports what it found; the
+half-filled pair is refused by `report`, which is the command that acts on it,
+with the not-configured exit code and a sentence naming the file and the
+missing key. `init` refuses it earlier still, at the prompt that asked.
+
+**Everything else inside those tables is carried through untouched.** `submit`
+will want timeouts, category codes and whatever else each API turns out to
+need, and this addendum deliberately does not design that. `config.py` reads
+what decides whether a row exists and leaves the rest alone, exactly as it
+leaves other sections alone today.
+
+## `init` prompts
+
+`init` grows four questions: the MISP url and key, the AbuseIPDB key, the
+URLhaus key. All four are skippable, and they say so, because the relay
+question and the hop picker now say they are required and a user reads the
+difference.
+
+The MISP pair is validated together at the point it is asked, so answering a
+url and skipping the key re-asks rather than writing a broken section. That is
+this repository's standing prompt rule, learned from a hand test that found
+four instances of one mistake: an answer validated somewhere other than where
+it was given costs the whole run.
+
+The config carry-across already preserves sections `build()` does not produce,
+and `init.py`'s own tests cover a `[misp]` key surviving an init that only sets
+the relays. Those tests describe the behaviour this addendum now makes real
+rather than hypothetical.
+
+**These prompts are HAND-TESTED by the user, not unit-tested**, like every
+other prompt in the tool. Whether a question reads clearly has no assertion.
+
+**A key in the config is a secret at rest.** The file is already mode 0600 and
+so are its backups, which is why that was built before any key existed. No
+change, named here so it is not mistaken for an oversight.
+
+## Testing
+
+Tested, because each has a right answer:
+
+- a configured destination with an acceptable IOC gets a row; one without an
+ acceptable IOC does not, and the URL-less case getting no `urlhaus` row is
+ the one to write first, since it is the rule's whole point
+- an unconfigured destination gets no row, however many IOCs would suit it
+- each row carries only the types its destination accepts, asserted against a
+ case holding an IP, a domain and a URL at once, which is the only shape that
+ can distinguish all three
+- MISP's row carries every IOC, including the domain neither vendor takes
+- a second `generate()` on the same manifest and config produces the same rows
+- the no-destination warning is written and the exit code is still zero
+- the config rejects a non-string key and a non-table section, each naming the
+ file
+- a half-filled MISP pair is refused by `report` with the not-configured exit
+ code, and does NOT stop `parse` from running, which is the half that would
+ regress silently
+
+Not tested: the `init` prompts, per the rule above.
+
+**No sweep is required for this change.** Both sweeps exist to prove that no
+recipient identifier reaches an IOC or a report body, and this addendum
+touches neither `parse.py` nor the content of any body. It adds rows to a
+manifest from values the user typed into their own config. Stated explicitly
+because `AGENTS.md` requires a sweep when `parse.py` changes, and a reader
+should be able to see that the question was asked rather than skipped.
+
+## What this leaves for `submit`
+
+Unchanged from the section above, plus one constraint this addendum creates:
+
+- **the `frozen` marker must be written in the same atomic manifest write as
+ the first `sent` status.** The spec above already required this; this
+ addendum now depends on it, because `generate()` rebuilds `destinations[]`
+ wholesale and the freeze is the only thing standing between that rebuild and
+ a manifest holding receipts.
+- the vendor payload shapes, per destination, against each vendor's real
+ documentation. URLhaus's needs an `auth.abuse.ch` account to read properly.
+- whether `report` learns to write those bodies, or `submit` does. The rows
+ now exist either way, which was the point of building them here.