diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/plans/2026-09-10-vendor-destinations.md | 1039 |
1 files changed, 1039 insertions, 0 deletions
diff --git a/docs/plans/2026-09-10-vendor-destinations.md b/docs/plans/2026-09-10-vendor-destinations.md new file mode 100644 index 0000000..751da23 --- /dev/null +++ b/docs/plans/2026-09-10-vendor-destinations.md @@ -0,0 +1,1039 @@ +# Vendor and MISP Destinations Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make `report` write the `misp` and `api` destination rows its spec has always required, so a case shows every destination it will reach before anything is sent. + +**Architecture:** A static table in `report.py` names each destination and the IOC types it accepts, transcribed from vendor documentation the way `init.PROVIDERS` is transcribed from SPF records. `config.py` learns which destinations are configured and reports that as data rather than raising. `report.generate()` appends vendor rows to the email rows it already builds. `init.py` and `cli.py` grow four skippable prompts. + +**Tech Stack:** Python 3.11+ stdlib only (`tomllib`, `unittest`). No new dependency: nothing here sends anything. + +**Read first:** `docs/specs/2026-09-09-report.md`, the addendum dated 2026-09-10 at the end. It settles every decision below and records why. `AGENTS.md` for the four non-negotiable properties. + +--- + +## Context an engineer needs before starting + +**This repository is TDD and the tests are the point.** Write the failing test, run it, watch it fail for the RIGHT reason, then implement. A test that passes before the implementation is a test that proves nothing. + +**Clear `__pycache__` between any mutation check.** A stale `.pyc` shadowed edited source in the last session and made a mutation check report nonsense. `python3 -B` does not help; it only stops bytecode being written. Use `find . -name __pycache__ -type d -exec rm -rf {} +`. + +**Run the suite with** `python3 -m unittest discover tests`. One module: `python3 -m unittest tests.test_report -v`. + +**Commits are GPG-signed.** `git commit -S`. Never `--no-verify`. Global hooks scan for personal data and a rejection is correct until proven otherwise. + +**Fixtures use `example.org`, `.invalid` and RFC 5737 documentation ranges only.** Never a real address, domain, or API key, in a test or a commit message. + +**What "configured" means throughout this plan:** a config section that is present AND carries every key it requires. `[abusedb]` and `[urlhaus]` require `api_key`. `[misp]` requires BOTH `url` and `api_key`; exactly one of the two is an error, not a skip, and Task 6 handles that separately from Task 3. + +**The five IOC type strings `parse.py` actually emits** are `ipv4`, `ipv6`, `domain`, `url` and `sha256`. Verified in `abusectl/parse.py:624`, `:631`, `:647` and `:658`. Do not invent a sixth. + +--- + +## File structure + +| File | Responsibility | Change | +|---|---|---| +| `abusectl/report.py` | the destination table, vendor row building, `generate()` | modify | +| `abusectl/config.py` | read which destinations are configured | modify | +| `abusectl/init.py` | render the three new sections, fix the footer | modify | +| `abusectl/cli.py` | the four prompts, the no-destination warning | modify | +| `tests/test_report.py` | vendor rows | modify | +| `tests/test_config.py` | configured-destination reading | modify | +| `tests/test_init.py` | rendering the new sections | modify | +| `tests/test_cli.py` | the warning, the MISP pair refusal | modify | + +No new files. The table belongs beside the code that consumes it, as `PROVIDERS` sits in `init.py`. + +--- + +### Task 1: The destination table + +**Files:** +- Modify: `abusectl/report.py` (add near the top, after the existing module constants) +- Test: `tests/test_report.py` + +- [ ] **Step 1: Write the failing test** + +Add to `tests/test_report.py`: + +```python +class DestinationTable(unittest.TestCase): + def test_every_destination_names_the_types_it_accepts(self): + self.assertEqual(report.DESTINATIONS["abusedb"]["accepts"], + ("ipv4", "ipv6")) + self.assertEqual(report.DESTINATIONS["urlhaus"]["accepts"], ("url",)) + self.assertEqual(report.DESTINATIONS["misp"]["accepts"], report.ALL_TYPES) + + def test_misp_is_its_own_kind_and_the_vendors_are_api(self): + self.assertEqual(report.DESTINATIONS["misp"]["kind"], "misp") + self.assertEqual(report.DESTINATIONS["abusedb"]["kind"], "api") + self.assertEqual(report.DESTINATIONS["urlhaus"]["kind"], "api") + + def test_all_types_covers_what_parse_emits(self): + # The five type strings parse.py writes. A sixth arriving without a + # decision about which destinations accept it would silently reach + # MISP only, so this test names them rather than deriving them. + self.assertEqual(report.ALL_TYPES, + ("ipv4", "ipv6", "domain", "url", "sha256")) +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `python3 -m unittest tests.test_report.DestinationTable -v` +Expected: FAIL with `AttributeError: module 'abusectl.report' has no attribute 'DESTINATIONS'` + +- [ ] **Step 3: Write minimal implementation** + +Add to `abusectl/report.py`, after the existing module-level constants: + +```python +# Every IOC type parse.py emits. Named rather than derived, so a new type +# arriving without a decision about which destinations accept it is a test +# failure rather than an indicator that quietly reaches MISP alone. +ALL_TYPES = ("ipv4", "ipv6", "domain", "url", "sha256") + + +# What each destination accepts, read from its own API documentation on +# 2026-09-10: +# +# misp MISP book, circl.lu/doc/misp/automation. Any attribute type. +# abusedb docs.abuseipdb.com, POST /api/v2/report: "a valid IPv4 or +# IPv6 address", and nothing else. +# urlhaus abuse.ch's own submit_url.py, POST urlhaus.abuse.ch/api/, +# whose submission array carries a url. The full docs are +# behind a login at auth.abuse.ch; re-verify the PAYLOAD there +# when submit is built. The accepted type is unambiguous. +# +# VirusTotal is deliberately absent: its only submission endpoints are +# POST /urls and file upload, so it accepts exactly what urlhaus accepts +# and takes no verdict with a submission. See backlog item 6. +# +# DO NOT EDIT THESE FROM MEMORY. init.PROVIDERS records what that costs: +# the first draft was written from memory and every range was wrong. A +# wrong "accepts" here means a row promising a submission the endpoint +# will refuse, or a missing row for something the vendor would have taken. +DESTINATIONS: dict[str, dict] = { + "misp": {"kind": "misp", "accepts": ALL_TYPES}, + "abusedb": {"kind": "api", "accepts": ("ipv4", "ipv6")}, + "urlhaus": {"kind": "api", "accepts": ("url",)}, +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `python3 -m unittest tests.test_report.DestinationTable -v` +Expected: PASS, 3 tests + +- [ ] **Step 5: Commit** + +```bash +git add abusectl/report.py tests/test_report.py +git commit -S -m "feat: add the destination table, transcribed from vendor docs" +``` + +--- + +### Task 2: Building vendor rows + +**Files:** +- Modify: `abusectl/report.py` (new function after `email_destinations`) +- Test: `tests/test_report.py` + +- [ ] **Step 1: Write the failing test** + +Add to `tests/test_report.py`: + +```python +class VendorDestinations(unittest.TestCase): + IOCS = [ + {"id": "ioc-1", "type": "ipv4", "value": "198.51.100.7"}, + {"id": "ioc-2", "type": "domain", "value": "example.invalid"}, + {"id": "ioc-3", "type": "url", "value": "http://example.invalid/a"}, + ] + + def test_a_row_carries_only_the_types_its_destination_accepts(self): + rows = report.vendor_destinations(self.IOCS, {"abusedb", "urlhaus"}) + by_id = {row["id"]: row for row in rows} + self.assertEqual(by_id["abusedb"]["iocs"], ["ioc-1"]) + self.assertEqual(by_id["urlhaus"]["iocs"], ["ioc-3"]) + + def test_misp_carries_every_ioc_including_what_no_vendor_takes(self): + rows = report.vendor_destinations(self.IOCS, {"misp"}) + self.assertEqual(rows[0]["iocs"], ["ioc-1", "ioc-2", "ioc-3"]) + + def test_a_destination_with_no_acceptable_ioc_gets_no_row(self): + # The rule's whole point: a case with no URL must not promise a + # urlhaus submission that would have nothing to submit. + no_urls = [ioc for ioc in self.IOCS if ioc["type"] != "url"] + rows = report.vendor_destinations(no_urls, {"abusedb", "urlhaus"}) + self.assertEqual([row["id"] for row in rows], ["abusedb"]) + + def test_an_unconfigured_destination_gets_no_row(self): + rows = report.vendor_destinations(self.IOCS, set()) + self.assertEqual(rows, []) + + def test_the_row_shape_is_the_shape_submit_iterates(self): + rows = report.vendor_destinations(self.IOCS, {"abusedb"}) + self.assertEqual(rows[0], { + "id": "abusedb", + "kind": "api", + "iocs": ["ioc-1"], + "body": None, + "status": "pending", + }) + + def test_no_row_carries_a_target(self): + # An endpoint is a property of the vendor, not of the case. Writing + # one into the evidence record would let a hand-edit redirect a + # submission somewhere the user never named. + rows = report.vendor_destinations(self.IOCS, {"misp", "abusedb"}) + for row in rows: + self.assertNotIn("target", row) + + def test_rows_come_out_in_table_order_not_set_order(self): + # The configured set is a set, so a stable order has to come from + # somewhere else or the manifest churns between runs. + first = report.vendor_destinations(self.IOCS, {"urlhaus", "misp", "abusedb"}) + second = report.vendor_destinations(self.IOCS, {"abusedb", "misp", "urlhaus"}) + self.assertEqual([row["id"] for row in first], + [row["id"] for row in second]) + self.assertEqual([row["id"] for row in first], + ["misp", "abusedb", "urlhaus"]) +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `python3 -m unittest tests.test_report.VendorDestinations -v` +Expected: FAIL with `AttributeError: module 'abusectl.report' has no attribute 'vendor_destinations'` + +- [ ] **Step 3: Write minimal implementation** + +Add to `abusectl/report.py`, after `email_destinations`: + +```python +def vendor_destinations(iocs: list[dict], configured: set) -> list[dict]: + """Build one row per configured destination that has something to send. + + BOTH conditions have to hold. A row for an unconfigured destination is a + promise that can only fail; a row for a configured one with nothing it + accepts is a promise with no content, an AbuseIPDB submission with + nothing to put in its `ip` parameter. + + Each row carries ONLY the IOCs its own destination accepts. A row is + what submit iterates, so a urlhaus row listing an IP is a submission + that gets built wrong or dropped at send time, whichever the + implementer notices first. + + Order comes from DESTINATIONS rather than from `configured`, which is a + set and therefore has no order worth writing into a manifest twice. + """ + rows = [] + for name, spec in DESTINATIONS.items(): + if name not in configured: + continue + accepted = [ioc["id"] for ioc in iocs if ioc.get("type") in spec["accepts"]] + if not accepted: + continue + rows.append({ + "id": name, + "kind": spec["kind"], + "iocs": accepted, + # Null until submit settles each payload shape. Writing a + # vendor's JSON now would mean guessing an endpoint's contract. + # No body_sha256 either: that hash records what was disclosed, + # and nothing has been. + "body": None, + "status": "pending", + }) + return rows +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `python3 -m unittest tests.test_report.VendorDestinations -v` +Expected: PASS, 7 tests + +- [ ] **Step 5: Commit** + +```bash +git add abusectl/report.py tests/test_report.py +git commit -S -m "feat: build vendor and MISP destination rows" +``` + +--- + +### Task 3: Reading which destinations are configured + +**Files:** +- Modify: `abusectl/config.py` (add to the `Config` dataclass and `load`) +- Test: `tests/test_config.py` + +**Read before starting:** the addendum's "Config" section. The MISP pair is NOT validated here; Task 6 does that in `report`. `config.load()` reports what it found. Raising here would make `parse` and `contacts` refuse to run over a section neither reads, and `cli.py` catches only `NotConfigured`, so the user would get a traceback. + +- [ ] **Step 1: Write the failing test** + +Add to `tests/test_config.py`: + +```python +class ConfiguredDestinations(unittest.TestCase): + def _load(self, text): + with tempfile.TemporaryDirectory() as tmp: + path = pathlib.Path(tmp) / "config.toml" + path.write_text( + '[general]\ntrusted_relays = ["192.0.2.0/24"]\n' + text + ) + return config.load(path) + + def test_no_sections_means_nothing_configured(self): + self.assertEqual(self._load("").destinations, set()) + + def test_a_vendor_with_a_key_is_configured(self): + settings = self._load('[abusedb]\napi_key = "k"\n') + self.assertEqual(settings.destinations, {"abusedb"}) + + def test_a_vendor_section_with_no_key_is_not_configured(self): + # Absent reads as not-configured, which is the whole convention. + self.assertEqual(self._load("[urlhaus]\n").destinations, set()) + + def test_an_empty_key_is_not_configured(self): + # api_key = "" is the trap this file exists to refuse: it reads as + # configured-and-broken and produces an auth error much later. + self.assertEqual(self._load('[urlhaus]\napi_key = ""\n').destinations, + set()) + + def test_misp_needs_both_halves(self): + both = '[misp]\nurl = "https://misp.example.invalid"\napi_key = "k"\n' + self.assertEqual(self._load(both).destinations, {"misp"}) + + def test_half_a_misp_pair_is_reported_separately_not_configured(self): + # Not an exception here: parse and contacts load this same file and + # read neither key. report refuses instead, with a sentence. + settings = self._load('[misp]\nurl = "https://misp.example.invalid"\n') + self.assertEqual(settings.destinations, set()) + self.assertEqual(settings.incomplete, {"misp": "api_key"}) + + def test_the_other_half_is_reported_too(self): + settings = self._load('[misp]\napi_key = "k"\n') + self.assertEqual(settings.destinations, set()) + self.assertEqual(settings.incomplete, {"misp": "url"}) + + def test_a_non_string_key_is_rejected_naming_the_file(self): + with self.assertRaises(ValueError) as caught: + self._load("[abusedb]\napi_key = 42\n") + self.assertIn("abusedb.api_key", str(caught.exception)) + self.assertIn("config.toml", str(caught.exception)) + + def test_a_non_table_section_is_rejected(self): + with self.assertRaises(ValueError) as caught: + self._load('abusedb = "nope"\n') + self.assertIn("abusedb", str(caught.exception)) +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `python3 -m unittest tests.test_config.ConfiguredDestinations -v` +Expected: FAIL with `AttributeError: 'Config' object has no attribute 'destinations'` + +- [ ] **Step 3: Write minimal implementation** + +In `abusectl/config.py`, add the required-keys table after `REPORTER_KEYS`: + +```python +# What each reporting destination's section must carry to count as +# configured. MISP takes a url because it is self-hosted and has no default +# endpoint; the vendors' endpoints are fixed and belong to their own +# destination modules. Everything ELSE inside these tables is carried +# through untouched: submit will want timeouts and category codes, and this +# reader deliberately does not design that. +DESTINATION_KEYS = { + "misp": ("url", "api_key"), + "abusedb": ("api_key",), + "urlhaus": ("api_key",), +} +``` + +Add two fields to the `Config` dataclass: + +```python + # Which destinations are fully configured, and which are half-filled. + # Reported as data rather than raised: every subcommand loads this file + # and most of them read neither key. + destinations: set = field(default_factory=set) + incomplete: dict = field(default_factory=dict) +``` + +Add to `load()`, before the `return`: + +```python + destinations = set() + incomplete = {} + for name, required in DESTINATION_KEYS.items(): + section = data.get(name) + if section is None: + continue + if not isinstance(section, dict): + raise ValueError( + f"{name} in {source} must be a table, got " + f"{type(section).__name__}" + ) + + present = [] + for key in required: + if key not in section: + continue + value = section[key] + # Rejected, not dropped, for the reason [reporter] records: a + # wrong value that reads as plausible is worse than an error. + if not isinstance(value, str): + raise ValueError( + f"{name}.{key} in {source} must be a string, got " + f"{type(value).__name__}: {value!r}" + ) + if value.strip(): + present.append(key) + + if len(present) == len(required): + destinations.add(name) + elif present: + # Some but not all: configured-and-broken rather than skipped. + # Absence of the WHOLE section is a skip; absence of one half of + # a pair is a mistake, and the command that acts on it says so. + missing = [key for key in required if key not in present] + incomplete[name] = missing[0] + + return Config(trusted_relays=trusted_relays, cases=cases, + reporter=reporter, destinations=destinations, + incomplete=incomplete) +``` + +Replace the existing `return Config(...)` line with the one above. + +- [ ] **Step 4: Run test to verify it passes** + +Run: `python3 -m unittest tests.test_config -v` +Expected: PASS, including the 9 new tests and every existing one + +- [ ] **Step 5: Commit** + +```bash +git add abusectl/config.py tests/test_config.py +git commit -S -m "feat: read which reporting destinations are configured" +``` + +--- + +### Task 4: Wiring vendor rows into `generate()` + +**Files:** +- Modify: `abusectl/report.py:1019` (`generate` signature and body) +- Modify: `abusectl/cli.py:445` (the `generate` call) +- Test: `tests/test_report.py` + +**The seam:** `generate()` takes the configured set as an ARGUMENT, the way it already takes the identity and `parse.py` takes the trust boundary. It must not read the config itself. That is what keeps the module pure and testable with no files on disk. + +- [ ] **Step 1: Write the failing test** + +Add to `tests/test_report.py`: + +```python +class GenerateWritesVendorRows(unittest.TestCase): + def _manifest(self): + return { + "format": 1, + "case_id": "2026-09-10-test", + "headers": [["From", "phish@example.invalid"], + ["Subject", "test"], + ["Date", "Wed, 09 Sep 2026 09:12:44 +0000"]], + "iocs": [ + {"id": "ioc-1", "type": "ipv4", "value": "198.51.100.7"}, + {"id": "ioc-2", "type": "url", + "value": "http://example.invalid/a"}, + ], + "contacts": [{"ioc": "ioc-1", "iocs": ["ioc-1"], + "abuse": ["abuse@example.invalid"]}], + } + + def test_configured_destinations_appear_alongside_the_email_rows(self): + with tempfile.TemporaryDirectory() as tmp: + out = report.generate(self._manifest(), pathlib.Path(tmp), + {"email": "r@example.org"}, + configured={"misp", "abusedb"}) + kinds = [row["kind"] for row in out["destinations"]] + self.assertIn("email", kinds) + self.assertIn("misp", kinds) + self.assertIn("api", kinds) + + def test_nothing_configured_leaves_the_email_rows_untouched(self): + with tempfile.TemporaryDirectory() as tmp: + out = report.generate(self._manifest(), pathlib.Path(tmp), + {"email": "r@example.org"}, + configured=set()) + self.assertEqual([row["kind"] for row in out["destinations"]], ["email"]) + + def test_a_second_run_produces_the_same_rows(self): + with tempfile.TemporaryDirectory() as tmp: + first = report.generate(self._manifest(), pathlib.Path(tmp), + {"email": "r@example.org"}, + configured={"misp"}) + ids = [row["id"] for row in first["destinations"]] + second = report.generate(first, pathlib.Path(tmp), + {"email": "r@example.org"}, + configured={"misp"}) + self.assertEqual([row["id"] for row in second["destinations"]], ids) + + def test_vendor_rows_get_no_body_written_to_disk(self): + with tempfile.TemporaryDirectory() as tmp: + out = report.generate(self._manifest(), pathlib.Path(tmp), + {"email": "r@example.org"}, + configured={"misp", "abusedb"}) + written = sorted(p.name for p in (pathlib.Path(tmp) / "bodies").iterdir()) + self.assertTrue(all(name.endswith(".xarf") for name in written)) + for row in out["destinations"]: + if row["kind"] != "email": + self.assertIsNone(row["body"]) + self.assertNotIn("body_sha256", row) + + def test_configured_defaults_to_nothing(self): + # An older caller that does not pass it must not crash, and must not + # silently gain destinations it never configured. + with tempfile.TemporaryDirectory() as tmp: + out = report.generate(self._manifest(), pathlib.Path(tmp), + {"email": "r@example.org"}) + self.assertEqual([row["kind"] for row in out["destinations"]], ["email"]) +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `python3 -m unittest tests.test_report.GenerateWritesVendorRows -v` +Expected: FAIL with `TypeError: generate() got an unexpected keyword argument 'configured'` + +- [ ] **Step 3: Write minimal implementation** + +In `abusectl/report.py`, change the `generate` signature: + +```python +def generate(manifest: dict, case_path: Path, identity: dict, + force: bool = False, configured: set | None = None) -> dict: +``` + +Add to the docstring, after the existing "ORDER MATTERS" paragraph: + +``` + `configured` is the set of destinations the config carries keys for, an + ARGUMENT rather than a config read, so this module stays pure and + testable with no files on disk. Defaulting to nothing configured is the + safe direction: a caller that forgets it writes no vendor row rather + than promising a submission nobody set up. + + destinations[] is rebuilt WHOLESALE every run, vendor rows included. + That is safe only while the freeze rule holds: once a destination has + landed the case is frozen and this function refuses to run at all. The + freeze marker is written by submit, so this is a constraint on that + spec, not merely a description of this one. +``` + +Replace the two lines at the end that build and assign destinations: + +```python + destinations = email_destinations(manifest.get("contacts", [])) + for destination in destinations: + text = build(manifest, destination, identity) + relative = f"bodies/{destination['id']}.xarf" + _write_body(case_path / relative, text) + destination["body"] = relative + destination["body_sha256"] = body_hash(text) + + destinations.extend( + vendor_destinations(manifest.get("iocs", []), configured or set()) + ) + manifest["destinations"] = destinations +``` + +Then in `abusectl/cli.py`, in `_cmd_report`, change the `generate` call: + +```python + manifest = report_module.generate( + manifest, args.case, identity, force=args.force, + configured=settings.destinations, + ) +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `python3 -m unittest discover tests` +Expected: PASS, everything + +- [ ] **Step 5: Mutation check** + +```bash +find . -name __pycache__ -type d -exec rm -rf {} + +cp abusectl/report.py /tmp/report.py.bak +``` + +Change `configured or set()` to `set()` in `generate`, then: + +```bash +find . -name __pycache__ -type d -exec rm -rf {} + +python3 -m unittest tests.test_report.GenerateWritesVendorRows -v +``` + +Expected: FAIL on `test_configured_destinations_appear_alongside_the_email_rows`. Restore with `cp /tmp/report.py.bak abusectl/report.py` and clear `__pycache__` again. **Restore from the copy, never with `git checkout`**, which would discard the whole file's work if anything else were uncommitted. + +- [ ] **Step 6: Commit** + +```bash +git add abusectl/report.py abusectl/cli.py tests/test_report.py +git commit -S -m "feat: write vendor rows into the manifest" +``` + +--- + +### Task 5: The no-destination warning + +**Files:** +- Modify: `abusectl/cli.py` (`_cmd_report`, after the identity check) +- Test: `tests/test_cli.py` + +**Warn and continue, exit zero.** `report` "exits non-zero only when it could not write", per the spec. A case with abuse desks and no vendors is a good report, and the tool is specified to be useful with no API key configured anywhere. + +- [ ] **Step 1: Write the failing test** + +Add to `tests/test_cli.py`, inside the existing report-command test class (the one with `_write_config`, `_make_case` and `_run`, around line 275). **Reuse those helpers; do not write new ones.** Note that `--config` is a GLOBAL option and goes BEFORE the subcommand. + +```python + def test_no_configured_destination_warns_and_still_exits_zero(self): + self._write_config('[reporter]\nemail = "r@example.org"\n') + case_path = self._make_case() + code, _, err = self._run( + "--config", str(self.config), "report", str(case_path) + ) + self.assertEqual(code, 0) + self.assertIn("no reporting destinations", err) + self.assertIn(str(self.config), err) + + def test_a_configured_destination_produces_no_warning(self): + self._write_config( + '[reporter]\nemail = "r@example.org"\n' + '\n[abusedb]\napi_key = "k"\n' + ) + case_path = self._make_case() + code, _, err = self._run( + "--config", str(self.config), "report", str(case_path) + ) + self.assertEqual(code, 0) + self.assertNotIn("no reporting destinations", err) +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `python3 -m unittest tests.test_cli.ReportWarnsWithNoDestinations -v` +Expected: FAIL on the first test, `AssertionError: 'no reporting destinations' not found in ''` + +- [ ] **Step 3: Write minimal implementation** + +In `abusectl/cli.py`, in `_cmd_report`, after the identity check and before `case.load`: + +```python + # Warned, not refused: a case with abuse-desk destinations and no + # vendors is a perfectly good report, and this tool is specified to be + # useful with no API key configured anywhere. Not silent either. A user + # who believes MISP is configured and finds no misp row has a typo'd + # section name, and MISP is the gate for everything irreversible: a case + # built with no gate, discovered at submit time, is discovered a step + # too late. + if not settings.destinations: + print( + "abusectl report: no reporting destinations configured, so this " + "case will reach abuse desks only. Add [misp], [abusedb] or " + f"[urlhaus] to {config_path}, or run `abusectl init`.", + file=sys.stderr, + ) +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `python3 -m unittest tests.test_cli -v` +Expected: PASS, everything + +- [ ] **Step 5: Commit** + +```bash +git add abusectl/cli.py tests/test_cli.py +git commit -S -m "feat: warn when no reporting destination is configured" +``` + +--- + +### Task 6: Refusing a half-filled MISP pair + +**Files:** +- Modify: `abusectl/cli.py` (`_cmd_report`, beside the warning from Task 5) +- Test: `tests/test_cli.py` + +**Refused by `report`, not by `config.load()`.** The test that matters most here is the second one: `parse` must still run. + +- [ ] **Step 1: Write the failing test** + +Add to the same report-command test class in `tests/test_cli.py`, reusing its helpers: + +```python + def test_a_misp_url_with_no_key_is_refused_naming_the_missing_key(self): + self._write_config( + '[reporter]\nemail = "r@example.org"\n' + '\n[misp]\nurl = "https://misp.example.invalid"\n' + ) + case_path = self._make_case() + code, _, err = self._run( + "--config", str(self.config), "report", str(case_path) + ) + self.assertEqual(code, 3) + self.assertIn("api_key", err) + self.assertIn(str(self.config), err) + + def test_half_a_pair_does_not_stop_parse(self): + # The half that would regress silently. Every subcommand loads this + # file; parse reads neither MISP key and must not be refused over + # one, nor traceback, since cli catches only NotConfigured. + self.config.write_text( + '[general]\ntrusted_relays = ["192.0.2.0/24"]\n' + f'cases = "{self.root / "cases"}"\n' + '\n[misp]\napi_key = "k"\n', + encoding="utf-8", + ) + code, _, _ = self._run( + "--config", str(self.config), "parse", + str(FIXTURES / "forged-chain.eml"), + ) + self.assertEqual(code, 0) +``` + +`forged-chain.eml` resolves against `trusted_relays = ["192.0.2.0/24"]`, which is what `tests/test_parse.py:42` uses for that fixture. The config here sets `cases` too, since `parse` writes one. + +- [ ] **Step 2: Run test to verify it fails** + +Run: `python3 -m unittest tests.test_cli.ReportRefusesHalfAMispPair -v` +Expected: FAIL on the first test with `AssertionError: 0 != 3` + +- [ ] **Step 3: Write minimal implementation** + +In `abusectl/cli.py`, in `_cmd_report`, BEFORE the no-destination warning from Task 5: + +```python + # Refused HERE rather than in config.load(), for the reason backlog + # item 5 already records one level up: an answer reported somewhere + # other than where it is used. Every subcommand loads this file, so + # raising in the reader would make `parse` refuse over a section it + # never reads, and cli catches only NotConfigured, so the user would + # get a traceback rather than a sentence. + if settings.incomplete: + for name, missing in sorted(settings.incomplete.items()): + print( + f"abusectl report: [{name}] in {config_path} is missing " + f"{missing}, so it is configured-and-broken rather than " + "skipped. Add it, or remove the section entirely.", + file=sys.stderr, + ) + return EXIT_NOT_CONFIGURED +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `python3 -m unittest discover tests` +Expected: PASS, everything + +- [ ] **Step 5: Commit** + +```bash +git add abusectl/cli.py tests/test_cli.py +git commit -S -m "feat: refuse a half-configured destination at report" +``` + +--- + +### Task 7: Rendering the new sections in `init.build()` + +**Files:** +- Modify: `abusectl/init.py` (`build`, `FOOTER`) +- Test: `tests/test_init.py` + +**The footer is wrong today.** It promises `[vendors]` as one table; the spec settles three separate sections. Fix it in this task or it documents a shape that never arrives. + +- [ ] **Step 1: Write the failing test** + +Add to `tests/test_init.py`: + +```python +class BuildsDestinationSections(unittest.TestCase): + ANSWERS = {"trusted_relays": ["192.0.2.0/24"]} + + def test_a_misp_pair_is_rendered(self): + text = init.build({**self.ANSWERS, + "misp_url": "https://misp.example.invalid", + "misp_api_key": "k"}) + data = tomllib.loads(text) + self.assertEqual(data["misp"]["url"], "https://misp.example.invalid") + self.assertEqual(data["misp"]["api_key"], "k") + + def test_a_vendor_key_is_rendered(self): + text = init.build({**self.ANSWERS, "abusedb_api_key": "k"}) + self.assertEqual(tomllib.loads(text)["abusedb"]["api_key"], "k") + + def test_a_skipped_destination_is_absent_not_empty(self): + # api_key = "" reads as configured-and-broken. Absent reads as + # not-configured and report can say so plainly. + text = init.build({**self.ANSWERS, "urlhaus_api_key": ""}) + self.assertNotIn("[urlhaus]", text) + self.assertNotIn("urlhaus", tomllib.loads(text)) + + def test_nothing_answered_renders_no_destination_section(self): + data = tomllib.loads(init.build(self.ANSWERS)) + for name in ("misp", "abusedb", "urlhaus"): + self.assertNotIn(name, data) + + def test_the_footer_no_longer_promises_a_vendors_table(self): + # It named [vendors] as one table; the spec settles three sections, + # and a footer describing a shape that never arrives is worse than + # no footer. + self.assertNotIn("[vendors]", init.build(self.ANSWERS)) + + def test_every_rendered_section_parses(self): + text = init.build({**self.ANSWERS, + "misp_url": "https://misp.example.invalid", + "misp_api_key": "k", + "abusedb_api_key": "a", + "urlhaus_api_key": "u"}) + data = tomllib.loads(text) + self.assertEqual(set(data) & {"misp", "abusedb", "urlhaus"}, + {"misp", "abusedb", "urlhaus"}) +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `python3 -m unittest tests.test_init.BuildsDestinationSections -v` +Expected: FAIL with `KeyError: 'misp'` on the first test + +- [ ] **Step 3: Write minimal implementation** + +In `abusectl/init.py`, replace `FOOTER`: + +```python +FOOTER = "\n".join([ + "", + "# Later parts of abusectl add further sections here:", + "# [reporting] - abuse-desk reporting defaults", +]) + "\n" +``` + +Add the destination field table after `_REPORTER_FIELDS`: + +```python +# Each destination's config keys and the answer key each is asked under. +# Rendered in this order so a written file is stable between runs. +_DESTINATION_FIELDS = ( + ("misp", (("url", "misp_url"), ("api_key", "misp_api_key"))), + ("abusedb", (("api_key", "abusedb_api_key"),)), + ("urlhaus", (("api_key", "urlhaus_api_key"),)), +) +``` + +In `build()`, before the final `return`: + +```python + # A section is emitted only when something was answered for it, the same + # rule [reporter] follows: an empty [abusedb] reads as + # configured-with-nothing, which is the api_key = "" trap wearing a + # different coat. + for section, fields in _DESTINATION_FIELDS: + rendered = [] + for key, answer_key in fields: + value = answers.get(answer_key, "") + if isinstance(value, str) and value.strip(): + rendered.append(f"{key} = {_quoted(value.strip(), key)}") + if rendered: + lines.extend(["", f"[{section}]"]) + lines.extend(rendered) +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `python3 -m unittest tests.test_init -v` +Expected: PASS, everything. + +The existing carry-across tests (`test_an_unsupplied_section_survives_a_rewrite` and `test_a_rewrite_still_loads`, `tests/test_init.py:261` and `:280`) still pass unchanged: both call `init.write` with only `trusted_relays`, so `build()` renders no `[misp]` and the old section is preserved exactly as before. The behaviour only changes when a run ANSWERS a MISP question, and then the new answer correctly wins, because `_rendered_sections` reads section names off the rendered text. + +- [ ] **Step 5: Commit** + +```bash +git add abusectl/init.py tests/test_init.py +git commit -S -m "feat: render the destination sections at init" +``` + +--- + +### Task 8: The four prompts + +**Files:** +- Modify: `abusectl/cli.py` (`_prompt_answers`, new `_ask_destinations`) +- Test: none. **These are hand-tested by the user**, per `AGENTS.md`: whether a question reads clearly has no assertion, and a test driving stdin asserts the wording it was written against and breaks on a rewording that improved it. + +- [ ] **Step 1: Write the implementation** + +In `abusectl/cli.py`, add after `_ask_reporter`: + +```python +def _ask_destinations() -> dict: + """Ask for the reporting destinations, all skippable. + + The MISP pair is validated TOGETHER and here, at the prompt that asked: + a url with no key is configured-and-broken rather than skipped, and + finding that out at `report` time costs the whole run. That is this + repository's standing prompt rule, learned from a hand test that found + four instances of one mistake. + + Skippable, and they say so, because the relay question and the hop + picker now say they are REQUIRED and a user reads the difference. + """ + print("\nabusectl can also file each case to a threat-intel platform.") + print("MISP is your own instance and is written FIRST: it is the record") + print("that gates everything irreversible, so a failure there stops the") + print("rest. The vendors are public feeds. Leave any blank to skip it.") + + answers = {} + + while True: + url = _ask("\nMISP instance URL (Enter to skip): ") + if not url: + answers["misp_url"] = "" + answers["misp_api_key"] = "" + break + if not url.startswith(("http://", "https://")): + print(" Expected a URL such as https://misp.example.org.") + continue + key = _ask("MISP API key (Enter to skip MISP entirely): ") + if not key: + # Half a pair is worse than none: it reads as configured and + # fails at submit. Ask the pair again rather than writing it. + print(" A URL with no key cannot be used, so MISP is skipped.") + answers["misp_url"] = "" + answers["misp_api_key"] = "" + break + answers["misp_url"] = url + answers["misp_api_key"] = key + break + + answers["abusedb_api_key"] = _ask( + "\nAbuseIPDB API key, for reporting IPs (Enter to skip): " + ) + answers["urlhaus_api_key"] = _ask( + "URLhaus Auth-Key, for reporting URLs (Enter to skip): " + ) + + if not any(answers.values()): + print("\n No destinations set. Cases will reach abuse desks only,") + print(" which is a complete report; add them later if you want one.") + + return answers +``` + +Change `_prompt_answers` to include them: + +```python + return {"trusted_relays": trusted_relays, "cases": cases, + **_ask_reporter(), **_ask_destinations()} +``` + +- [ ] **Step 2: Verify the suite still passes** + +Run: `python3 -m unittest discover tests` +Expected: PASS, everything. `tests/test_cli.py` covers dispatch, not prompts. + +- [ ] **Step 3: Hand-test, and ask the user to run it** + +```bash +python3 -m abusectl init --config /tmp/handtest.toml +``` + +Walk every path: skip everything; answer MISP fully; answer a MISP url then skip the key (must skip MISP, not write half); type a bare hostname for the MISP url (must re-ask). Then `cat /tmp/handtest.toml` and confirm no `api_key = ""` anywhere, and `python3 -c "import tomllib; tomllib.load(open('/tmp/handtest.toml','rb'))"` parses. + +**Ask the user to do this pass themselves.** The last hand test found five defects the suite could not, four of them the same mistake. + +- [ ] **Step 4: Commit** + +```bash +git add abusectl/cli.py +git commit -S -m "feat: ask for the reporting destinations at init" +``` + +--- + +### Task 9: Documentation + +**Files:** +- Modify: `AGENTS.md` (the architecture block and the config section) +- Modify: `README.md` (config example, if it carries one) + +- [ ] **Step 1: Update AGENTS.md** + +In the "Config" section, after the two traps, add: + +```markdown +**A reporting destination is configured only when its section carries every +key it needs.** `[abusedb]` and `[urlhaus]` need `api_key`; `[misp]` needs +`url` AND `api_key`. Exactly one half of the MISP pair is +configured-and-broken rather than skipped, and `config.load()` reports it in +`incomplete` rather than raising: every subcommand loads this file, and +`parse` must not be refused over a section it never reads. `report` refuses +it, because that is the command that acts on it. +``` + +In the architecture block, extend the `report.py` line: + +``` + report.py IOCs + contacts -> bodies + destinations[] pure +``` + +And add after the block: + +```markdown +**`report.DESTINATIONS` is transcribed from vendor documentation**, read +2026-09-10, the same rule as `init.PROVIDERS`: do not edit it from memory. A +wrong `accepts` means a row promising a submission the endpoint will refuse, +or a missing row for something the vendor would have taken. VirusTotal is +deliberately absent; backlog item 6 records the findings. +``` + +- [ ] **Step 2: Check the README** + +```bash +grep -n 'trusted_relays\|\[reporter\]\|config.toml' README.md +``` + +If it shows a config example, add the three sections to it as commented-out optional entries. If it does not, skip this step. + +- [ ] **Step 3: Verify the suite one final time** + +```bash +find . -name __pycache__ -type d -exec rm -rf {} + +python3 -m unittest discover tests +``` + +Expected: OK, with the new tests counted. Note the number: it was 366 before this plan. + +- [ ] **Step 4: Commit** + +```bash +git add AGENTS.md README.md +git commit -S -m "docs: record the destination table and the configured rule" +``` + +--- + +## What this plan does NOT do + +Named so nobody adds them unasked: + +- **No vendor payloads.** Bodies stay null. The shapes belong to the `submit` spec, against each vendor's real documentation, and URLhaus's needs an `auth.abuse.ch` account to read properly. +- **No network.** Nothing here opens a socket. `tests/test_offline.py` enumerates modules by import and must keep passing unchanged. +- **No sweep.** Both sweeps prove no recipient identifier reaches an IOC or a body. This touches neither `parse.py` nor body content; it adds rows from values the user typed into their own config. Stated because `AGENTS.md` requires a sweep when `parse.py` changes, and a reader should see the question was asked. +- **No `submit`, no `retry`, no freeze marker.** The freeze is named as a constraint on the submit spec in the addendum, and stays there. |