1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
|
# abusectl `report`: IOCs and contacts to report bodies
Status: **agreed 2026-09-09**, in one brainstorming session with the user.
This spec settles the part the umbrella design left open, named there as
"the X-ARF (RFC 5965) schema version, which fields the user's reporting
identity fills, and the plain-text alternative for desks that do not parse
X-ARF".
Read `docs/specs/2026-09-08-abusectl-design.md` first. This document assumes
its manifest format, its case directory and its ordering between the parts.
It changes one thing there and says so where it does: the re-run guard is a
content hash rather than a timestamp.
## What it does
```
abusectl report <case>
```
Reads a case manifest, groups the resolved contacts into destinations, writes
a report body per destination under `bodies/`, and rewrites the manifest with
a `destinations[]` array. It is OFFLINE and PURE: it opens no socket, sends
no mail, and reads no file outside the case directory except the config.
It is the last step before anything irreversible happens. What it produces is
a document the user reads, edits and approves, so its output is written for a
human first and a parser second.
## Modules
```
abusectl/
report.py IOCs + contacts -> bodies + destinations[] pure
```
One module. There is no protocol/policy split here of the kind that puts
`rdap.py` beside `contacts.py`, because there is no protocol: X-ARF is a MIME
document and `email.message` in the standard library already is that layer.
`report.py` takes the reporting identity as an ARGUMENT, the way `parse.py`
takes the trust boundary. `cli.py` reads the config and passes it in. That is
what keeps the module testable with no files on disk, and it matters more here
than it did for `parse`: the identity is the one thing in a report that is
disclosed deliberately, and a module that reaches for it itself is a module
that can disclose it in a code path nobody reviewed.
## The report
Each email destination gets one MIME document, `multipart/report` with
`report-type=feedback-report`, per RFC 5965. Three parts, in order.
### Part 1, `text/plain`: what a human reads
This is the part that decides whether the report is acted on. Desks triage a
queue; a report whose ask is buried is a report that waits.
```
Phishing message received 2026-09-08, reporting infrastructure on your
network. Requesting takedown and customer notification.
Observed on your infrastructure:
198.51.100.7 sending IP, first hop outside our trust boundary
example.invalid domain in message links, via a.b.c.example.invalid
Message as declared:
Date: Mon, 08 Sep 2026 09:12:44 +0000
From: "Example Bank" <phish@example.invalid>
Subject: Your account requires verification
Authentication results:
SPF: fail DKIM: none DMARC: fail
URLs, redacted:
http://login-example.invalid/verify?id=REDACTED&src=REDACTED
Recipient identifiers have been removed from this report by policy.
Parameter names are preserved, parameter values are not. Full evidence is
retained locally and is available on request.
Reported by: Danilo M., Example Consulting <reporter@example.org>
Generated by abusectl/<version>.
```
Four decisions in that shape.
**The ask is the first sentence.** Not the evidence, not the identity. A desk
reading one line must know what happened and what is wanted.
**Only the recipient's own indicators appear.** Destinations are grouped per
abuse address (below), and a desk shown three IPs that are not theirs stops
reading. The `queried` field from `contacts` earns its keep here: "via
`a.b.c.example.invalid`" tells the desk why they are being mailed about a name
that is not literally in the message.
**The redaction note is ALWAYS present, never conditional on whether anything
was redacted.** A desk that sees `?id=REDACTED` with no explanation may read
the report as malformed or doctored. One sentence turns that into a report
that looks careful, and it opens the door for a desk that genuinely needs more
to ask for it, which is the hand-paste route during review.
**Plain text, hard-wrapped at 72 columns, with no HTML alternative.** Abuse
desks run ticketing systems and many strip HTML. An HTML part would be a
second body to keep in sync with the first for no reader.
### Part 2, `message/feedback-report`: what a parser reads
An RFC 5965 envelope carrying x-arf fields inside it.
```
Feedback-Type: fraud
User-Agent: abusectl/0.1.0
Version: 1
Source: 198.51.100.7
Source-IP: 198.51.100.7
Reported-Domain: example.invalid
Arrival-Date: Mon, 08 Sep 2026 09:12:44 +0000
Reported-URI: http://login-example.invalid/verify?id=REDACTED
```
`Feedback-Type`, `User-Agent` and `Version` are the three fields RFC 5965
requires. The rest are optional there or come from x-arf.
**Every field name here is in the IANA MARF registry except `Source`**, which
is x-arf's and is the reason the envelope carries x-arf fields at all. An
earlier draft of this example also carried `Report-Type: phishing`, which is
not registered anywhere; it was removed once the registry was checked, since
`Feedback-Type` already says it. `fraud` rather than `abuse` because RFC 5965
defines it as "some kind of fraud or phishing activity", which is what this
tool reports. See backlog item 4 for what was checked against what.
**Why an RFC 5965 envelope with x-arf fields inside, rather than either
alone.** RFC 5965 is an IETF standard and universally understood, but it was
designed for feedback loops, where a report is ABOUT A MESSAGE. These reports
are about INDICATORS, and 5965 has no natural field for "this specific host is
the thing being reported". x-arf's `Source` does. The envelope is the
standard's own extension point: the part is key/value, so a 5965 parser reads
the fields it knows and ignores the rest, and x-arf tooling finds what it
wants.
**This choice deliberately does not depend on which of the two is more widely
deployed**, which is a number nobody publishes and which this document does
not claim to know. It was chosen so that the answer does not matter: a
standards parser works, x-arf tooling works, and the human part works
regardless of both.
**`Source` is singular and a destination may carry several indicators.** The
primary indicator fills it; the full list appears in the text part and in
repeated `Reported-Uri` and `Source-IP` fields. RFC 5965 permits one report
part per indicator instead, and that was rejected as heavier for no reader:
the desk acts on the incident, not on each row.
### Part 3, `text/rfc822-headers`: the message itself, almost
**The original message is NOT attached.** `source.eml` carries every
identifier the first property exists to keep out: `To`, `Cc`, `Delivered-To`,
unredacted URLs whose query and path segments encode the recipient, the user's
own Message-IDs, maildir paths and account keys. An abuse desk forwards a
report to the abused customer, and for a phishing domain that customer may be
the attacker; URLhaus is a public feed. Attaching it would deanonymise the
reporter to the attacker, and for a consultant the tracking parameter may
carry a CLIENT's identifier rather than the user's own.
RFC 5965 provides `text/rfc822-headers` for exactly the case where the full
message cannot be included, so this is the standard's own answer and not a
deviation from it.
The headers included are a WHITELIST:
```
Received (down to the untrusted hop only, never below)
From, Subject, Date, Message-ID, Reply-To, Return-Path
Authentication-Results, Received-SPF
MIME-Version, Content-Type
```
**A whitelist, never a blacklist.** A blacklist means every header the parser
learns to read later is a leak waiting for someone to remember. This is the
same reasoning that has `parse.py` not reading `To` at all rather than
stripping it afterwards.
`Subject` and the `From` display name are attacker-controlled free text, and a
sweep has already found a spoofed `Reply-To` display name. They are kept: they
are the message's own content rather than the recipient's identity, and they
are what lets a desk recognise a campaign they have seen before.
**Considered and not built: a redacted body text part.** A desk analysing a
campaign wants the lure, the impersonated brand and the pretext, and none of
that survives headers-only. The body's INDICATORS already survive as IOCs
regardless, so what is lost is the prose. It is not built because the prose is
an unbounded attacker-supplied string, and deciding what is safe inside free
text is a judgement rather than a whitelist, which is the shape of every leak
this project has had. Build it when a desk actually asks for the lure, and
build it as a redaction rule with its own tests, not as a passthrough.
## Where the headers come from
**`parse` stores the whitelisted headers in the manifest, and `report` never
opens `source.eml`.**
This is the same structural argument as the first property, applied one level
down: `report` cannot disclose a header it was never given. The alternative,
re-reading `source.eml` at report time and filtering there, would put a second
"what may be disclosed" decision in a second module, away from `parse.py`
where that decision currently lives, and two places to remember is how the
fourth property leaked three times.
The cost is real and is accepted: this is a change to `parse.py`, a new
`headers` block in the manifest, and **both sweeps must be re-run**, per
`AGENTS.md`. The sweep assertion is what proves the whitelist does not carry
an address, and a whitelist written by hand is exactly the kind of thing a
sweep catches being wrong.
### Two accepted disclosures, named so they are not mistaken for leaks
The first property reads as an unqualified "recipient identifiers must never
reach a report". These are the deliberate exceptions the whitelist creates,
recorded here rather than left to be rediscovered in a test comment.
**Our own receiving relay's hostname is published.** The boundary `Received`
line names it in its `by` clause and `Authentication-Results` names it as the
authserv-id, so `mx.example.org` travels with every report. That is the
user's mail host, not the user's identity, and an abuse desk learns it from
the report's own `From` regardless. It is accepted because removing it would
mean rewriting the inside of two headers whose value to a desk is precisely
that they are the receiving server's own verbatim words. The consequence is
that the manifest-wide "no bare `example.org`" assertion cannot hold over the
`headers` block; `tests/test_cli.py` narrows it there and asserts the
ADDRESS is still absent, which is the part that matters.
**Attacker-controlled free text is published unfiltered.** `Subject` and the
`From` display name are kept deliberately, because they are what lets a desk
recognise a campaign. An attacker who writes the recipient's address into
one, plainly or obfuscated as `you%40example.org`, gets it published: the
whitelist governs WHICH headers travel, never what is inside one. This is
not fixed by filtering free text, which is the judgement-shaped problem that
`AGENTS.md` names as the origin of every leak this project has had. The sweep
over real mail is what covers this class, which is one more reason it is not
optional here.
The envelope recipient is NOT in this list. Our own relay writes it into the
boundary `Received` line's optional `for` clause, and that clause is cut
before the line is stored, in every shape the grammar allows.
## The reporting identity
Three config keys, all under a `[reporter]` section:
```toml
[reporter]
name = "Danilo M."
org = "Example Consulting"
email = "reporter@example.org"
```
They fill the report's `From`, the `Reported by:` line in the text part, and
nothing else. `User-Agent` is `abusectl/<version>` and is not configurable.
**The reporter's identity is disclosed DELIBERATELY, and that is what makes it
different from every other identifier this tool refuses to publish.** The same
address recovered from a `To` header is a leak; supplied in a config file it
is the user choosing to be identified, and a report with no reply address is
one a desk deprioritises. The distinction is provenance, so it is enforced by
provenance: the identity comes from config ONLY, and `parse` must never supply
it. If the two ever became one path, the distinction would be a comment rather
than a guarantee.
A skipped answer is ABSENT from the config, never an empty string, the same
rule the rest of the config follows. `init` grows prompts for these three, and
they are hand-tested like the rest of the prompts.
**Noted, not built: a per-case reporting identity.** Reporting a campaign that
targeted a client, under the user's own name, tells the abuse desk which
consultant is working that incident, which is a disclosure about the
engagement rather than about the mail. The escape hatch already exists without
new machinery: review edits the bodies, and a `From` in a body is a line the
user can change. A config-level override belongs to the first real engagement
where it matters, not to this spec.
## Destinations
**One destination per abuse ADDRESS.** Every IOC whose contact resolved to
`abuse@example.invalid` is grouped into one report to that desk.
Contacts already fold by host, but two different contacts can still resolve to
the same address, an IP and a domain both at one hoster being the common case.
Grouping per contact would send that desk two mails about one incident, which
is the duplicate-mail behaviour desks complain about. Grouping per IOC would
be a storm.
`report` writes three kinds of destination:
| kind | body | who builds the payload |
|---|---|---|
| `email` | `bodies/<id>.xarf` | `report`, the MIME document above |
| `api` | `null` for now, see below | `submit`, once its spec settles the shape |
| `misp` | `null` | `submit`, via PyMISP |
MISP is a destination like the others with a null body, per the umbrella
design, so `submit` stays one loop with one ordering rule.
**Only `email` bodies are built by THIS spec.** The `api` kinds are created as
destination entries with their IOC lists and a `pending` status, and their
bodies stay null until the `submit` spec settles each vendor's payload shape.
Writing a vendor's JSON now would mean guessing an endpoint's contract from
memory, which is the mistake the provider table already records: the first
draft was written from memory and every range was wrong. The destination
entries exist from the start so that `submit` fills bodies rather than
inventing rows, and so review can already see which vendors a case will reach.
### An indicator with no abuse contact
A missing contact is a normal outcome, not an error: the umbrella design
already settles that, and RDAP publishes no abuse role for many netblocks.
`report` writes an `unreportable[]` array into the manifest, one entry per IOC
that reached no email destination, carrying the reason from its contact entry:
```json
"unreportable": [
{ "ioc": "ioc-4", "reason": "no abuse role published" },
{ "ioc": "ioc-9", "reason": "not ASCII, and we do not guess at an IDN encoding" }
]
```
Two reasons for making it explicit. Review becomes honest: the user sees that
four indicators are going to MISP and the vendors but no desk was found for
them, which is a fact they may want to act on by hand, and finding it any other
way means diffing the IOC list against every destination's IOC list. And it is
the same instinct as `suspect_path_segments` flagging rather than redacting: a
failure that is visible beats a failure that is merely absent.
**It is not an error and does not affect the exit code.** A case where nothing
resolved still produces MISP and vendor destinations and is a perfectly good
report. `report` exits non-zero only when it could not write.
## Re-running, and the frozen case
The umbrella design says `report` refuses to run again on a case whose bodies
were modified after generation, by a timestamp check. **The intent stands and
the mechanism changes**: it is a content hash.
An mtime is a poor witness in both directions. A `git checkout`, an `rsync`, a
backup restore or an editor that writes-and-renames all move mtime with no
human having edited anything, and an editor that preserves mtime hides a real
edit. So `report` records the SHA-256 of each body it writes, in that body's
destination entry, and compares content rather than a rumour about content.
`hashlib` is standard library, so this costs a field and no dependency.
The hash has a SECOND job, and the spec states it so a later change does not
drop it as redundant: for a destination that has been sent, the hash is the
record of what was actually disclosed to a third party.
### The rule
**A case where ANY destination has been sent is FROZEN.** `report` refuses,
and there is no `--force` override.
The destinations are not independent artifacts, they are one incident reported
in parallel. If one desk holds the report and a body for another desk is then
regenerated with different content, two desks hold contradictory accounts of
the same case, and a desk that forwards to the other finds the reporter
unreliable. Regeneration is also not as isolated as it looks: the shared parts,
the identity, the header block, the IOC list, come from the manifest, so
regenerating one body after the manifest has changed produces a case whose
bodies were built from two different states.
Freezing is recorded EXPLICITLY, written by `submit` at its first success:
```json
"frozen": { "at": "2026-09-08T12:40:11Z", "by": "abusedb" }
```
Absent means not frozen, the same convention the config follows for a skipped
answer. It is an optional field, so existing manifests stay loadable and the
`format` version does not change.
**Explicit rather than derived from the statuses**, because the marker is
write-once and monotonic. A status corrected by hand, or a status added by a
later schema that nobody remembered to add to a frozen set, would quietly
unfreeze a derived check. For a rule protecting an evidence record, a field
that can only be turned on is the right shape.
`submit` must write the marker in the SAME atomic manifest write as the first
`sent` status. A separate write leaves a window where a crash produces a case
that has been disclosed and does not know it.
### The whole rule, in order
| case state | `report` does |
|---|---|
| `frozen` present | refuses, no override, names the destination that landed |
| bodies modified, not frozen | refuses without `--force` |
| bodies modified, `--force` | backs each up to `<name>.<timestamp>.orig`, regenerates |
| bodies missing or unmodified | regenerates |
A deleted body regenerates silently. The only reasons to delete one are a
mistake or a deliberate start-over, and regeneration is what both want; the
case that looked like it needed protecting, a body deleted after it was sent,
is caught by the freeze rather than by the file check.
Because `--force` can now only ever touch a case that nothing has left, it is
a far safer flag than it first appears. The backup is kept anyway: the
umbrella design's point about a review that took twenty minutes applies, and
`init` already backs up a config it is about to replace.
A user who needs to change a body on a frozen or deferred case still can, by
editing it during review. That route is unaffected and is the right one: it is
a deliberate act with the user looking at the text.
## Testing
TDD, and the same rule as the rest of the repository: test what has a right
answer.
Tested, because there is one:
- the MIME structure: three parts, the right types, `report-type=feedback-report`
- the header whitelist keeps what it should and, more importantly, DROPS
`To`, `Cc`, `Delivered-To` and `X-Original-To` when a fixture carries them
- `Received` is truncated at the untrusted hop and never includes the ones
below it, against `forged-chain.eml`
- destinations group per address, including the two-contacts-one-address case
- an IOC with no contact lands in `unreportable[]` and creates no destination
- the body hash detects a modified body, and does not fire on an untouched one
- a frozen case refuses even with `--force`
- `--force` writes the timestamped backup before overwriting
Not unit-tested: whether the text part READS well to an abuse desk. That has
no assertion, and it is the same category as the interactive prompts. The user
hand-tests it by reading a generated report.
**The leak sweep must be re-run**, both A and B, because this spec changes
`parse.py`. Sweep A gains a third assertion: no address from the raw source
appears in any generated report body. That is the assertion that actually
proves the whitelist, and it should be as broad as the existing one, every
address in the source against every byte of every body.
## What this leaves for `submit`
Named here because this spec creates them, not to settle them:
- writing the `frozen` marker atomically with the first `sent` status
- the vendor JSON shapes, and which side writes them. `report` creates the
`api` destination rows; whether it also learns to write their bodies is a
decision for that spec. If a vendor's payload turns out to need a value the
manifest does not hold, that is a change here, not a workaround there.
---
# Addendum, agreed 2026-09-10: the vendor and MISP destinations
Status: **agreed 2026-09-10**, in one brainstorming session with the user.
This addendum completes the spec above rather than superseding it. Everything
before this line stands as written; what follows settles the part of the
"Destinations" section that was specified and never built.
## Why this exists
The destination table above names three kinds, `email`, `api` and `misp`, and
says the `api` and `misp` rows are "created as destination entries with their
IOC lists and a `pending` status" so that "review can already see which vendors
a case will reach".
**`report` writes only `email` rows.** Verified through the public API on
2026-09-10: `generate()` on a manifest with one IP contact returns
`destinations` containing exactly one row, of kind `email`. Nothing in the
module or the suite mentions a `misp` or `api` row; the only occurrences of
`misp` in the tree are `init.py`'s config carry-across.
This was not a defect in what was built. The implementation plan
(`docs/plans/2026-09-09-report.md`) carried thirteen tasks and none of them was
this one, so the module is complete against its plan and incomplete against its
spec. The gap was found when the `submit` brainstorm began, because `submit`
is specified to fill bodies into rows that do not exist.
The consequence while it stands is that review is less honest than the spec
promised: the user approving a case sees the abuse desks it will mail and no
indication that the same case is bound for MISP and two vendors.
## Which destinations, and what each accepts
Three destinations. VirusTotal was considered and is deferred; see backlog
item 6 for the findings, which are worth reading before it is reconsidered.
The accepted types below were read from each vendor's own API documentation on
2026-09-10, not written from memory. That distinction is the provider table's
lesson (`init.PROVIDERS`): the first draft of that table was written from
memory and every single range was wrong.
| destination | accepts | auth header | endpoint | source |
|---|---|---|---|---|
| `misp` | every IOC type | `Authorization` | `POST /events`, `/attributes/add/` | MISP book, circl.lu/doc/misp/automation |
| `abusedb` | `ipv4`, `ipv6` | `Key` | `POST /api/v2/report` | docs.abuseipdb.com |
| `urlhaus` | `url` | `Auth-Key` | `POST https://urlhaus.abuse.ch/api/` | abuse.ch's own `submit_url.py` |
The endpoints and headers are recorded here because they were checked in the
same pass, not because `report` uses them. **`report` uses only the "accepts"
column.** Resolving an endpoint is `submit`'s, and the table is here so that
spec starts from a checked source rather than re-deriving one.
**URLhaus's full submission documentation is behind a login at
`auth.abuse.ch`.** The row above comes from the reference `submit_url.py` that
abuse.ch publishes, which is a primary source but a narrower one than the docs.
Re-verify the payload against the real documentation when the user has an
account; that is a `submit` task, since only the accepted TYPE matters here and
that much is unambiguous.
## The rule: configured AND acceptable
A destination earns a row when **both** hold:
1. it is configured, meaning its config section is present and carries every
key that section requires (`api_key` for the two vendors, both `url` and
`api_key` for MISP; a section present with only one half of the MISP pair
is an error rather than a skip, see Config below), and
2. the case holds at least one IOC of a type it accepts.
Either alone is the wrong rule. A row for an unconfigured vendor is a promise
that can only fail. A row for a configured vendor with nothing it can take is
a promise with no content: an AbuseIPDB submission for a case whose only
indicators are URLs has nothing to put in its `ip` parameter.
**Each row carries only the IOCs its own destination accepts**, not the case's
whole list. A row is what `submit` iterates, and a row listing a URL under
AbuseIPDB is a submission that will be built wrong or dropped at send time,
whichever the implementer notices first. MISP's row therefore carries every
IOC and the vendor rows carry subsets, which is also what makes the ordering
rule legible during review: MISP holds the whole incident, the vendors hold
the parts they can act on.
## The row shape
```json
{ "id": "misp", "kind": "misp", "iocs": ["ioc-1","ioc-2","ioc-3"],
"body": null, "status": "pending" },
{ "id": "abusedb", "kind": "api", "iocs": ["ioc-1"],
"body": null, "status": "pending" },
{ "id": "urlhaus", "kind": "api", "iocs": ["ioc-3"],
"body": null, "status": "pending" }
```
The same keys as an email row, so `submit` stays one loop with one ordering
rule, as the umbrella design requires.
**The id is the destination's name, a fixed literal, not a derived hash.** An
email destination hashes its target address because a desk is discovered per
case and its address is the only stable thing about it. There is exactly one
AbuseIPDB. A literal id is also what lets `retry` find a row across cases
without knowing how it was built.
**There is no `target`.** An email row carries the abuse address it will mail.
A vendor's endpoint is a property of the vendor rather than of the case, and
writing it into the manifest would put a submission URL inside the evidence
record, where a hand-edit or a corrupted file could redirect a submission to
somewhere the user never named. `submit` resolves the endpoint from the
destination module.
**`body` stays null and there is no `body_sha256`.** The spec above defers the
vendor payload shapes to `submit`, and the hash exists to record what was
disclosed. Neither has anything to record yet. Writing a vendor's JSON now
would mean guessing an endpoint's contract, which is the mistake this document
already refuses once.
## Rebuilding, and what makes it safe
**`generate()` rebuilds `destinations[]` wholesale on every run**, email rows
and vendor rows alike. There is no status-preserving merge and no matching by
id.
Today the module already assigns over the key, which was found during this
session to discard a hand-constructed row marked `sent`. That is not a live
defect: nothing has ever sent, because `submit` does not exist, so no manifest
anywhere carries a status other than `pending` and there is no data to
preserve.
**This is safe only while the freeze rule holds**, and that dependency is
stated rather than left implicit. Once a destination has landed, the case is
frozen and `report` refuses to run at all, so a rebuild can never reach a
manifest holding a receipt. The freeze marker is written by `submit`, which is
unbuilt, so this is a constraint on that spec and not merely a description of
this one: **if `submit` ships without writing `frozen` atomically with its
first `sent` status, this rebuild becomes a data-loss bug.** It is named again
in the handover section below.
A status-preserving merge was considered and rejected. It would be a second
mechanism enforcing what the freeze already enforces, and two mechanisms for
one rule is how a derived check quietly comes to disagree with the explicit
one. The spec above rejected a derived freeze check for the same reason.
## Nothing configured
`report` **warns and continues.** It writes a line to stderr naming the config
file and saying no reporting destinations are configured, then exits zero.
Not an error, on the spec's own rule that `report` "exits non-zero only when it
could not write". A case with abuse-desk destinations and no vendors is a
perfectly good report, and the tool is specified to be useful with no API key
configured anywhere: `parse` + `contacts` + `report` produces a document that
can be sent by hand.
Not silent either, because the failure it hides is the one worth catching. A
user who believes MISP is configured and finds no `misp` row has a typo'd
section name, and the umbrella design makes MISP the gate for everything
irreversible. A case built with no gate, discovered at submit time, is
discovered a step too late.
## Config
`config.py` grows one more thing it reads: which destinations are configured.
```toml
[misp]
url = "https://misp.example.invalid"
api_key = "..."
[abusedb]
api_key = "..."
[urlhaus]
api_key = "..."
```
MISP takes a `url` because it is self-hosted and has no default endpoint. The
two vendors take a key only; their endpoints are fixed and belong to their
destination modules.
The rules are the ones the file already follows, applied unchanged. A skipped
answer is ABSENT, never an empty string. A non-string value is REJECTED with a
message naming the file, not silently dropped, on the same reasoning that
governs `[reporter]`: a value that reads as plausible and is wrong is worse
than an error, and `ipaddress.ip_network(42)` is the trap this file learned it
from. A section that is not a table is rejected the way `reporter = "A
Reporter"` already is.
**MISP's two keys are validated as a PAIR, and WHERE that is reported
matters.** A url with no key, or a key with no url, is configured-and-broken
rather than not-configured. This is the one place the absent-means-skipped
convention needs help: absence of the whole section is a skip, absence of one
half of it is a mistake.
It must NOT be raised from `config.load()`. Every subcommand loads the config,
so a `ValueError` there would make `parse` and `contacts` refuse to run over a
half-filled MISP section that neither of them reads, and `cli.py` catches only
`NotConfigured`, so the user would get a traceback rather than a sentence. The
same trap the file already records for `[reporter]`, one level up: an answer
reported somewhere other than where it is used.
So `config.load()` reads the section and reports what it found; the
half-filled pair is refused by `report`, which is the command that acts on it,
with the not-configured exit code and a sentence naming the file and the
missing key. `init` refuses it earlier still, at the prompt that asked.
**Everything else inside those tables is carried through untouched.** `submit`
will want timeouts, category codes and whatever else each API turns out to
need, and this addendum deliberately does not design that. `config.py` reads
what decides whether a row exists and leaves the rest alone, exactly as it
leaves other sections alone today.
## `init` prompts
`init` grows four questions: the MISP url and key, the AbuseIPDB key, the
URLhaus key. All four are skippable, and they say so, because the relay
question and the hop picker now say they are required and a user reads the
difference.
The MISP pair is validated together at the point it is asked, so answering a
url and skipping the key re-asks rather than writing a broken section. That is
this repository's standing prompt rule, learned from a hand test that found
four instances of one mistake: an answer validated somewhere other than where
it was given costs the whole run.
The config carry-across already preserves sections `build()` does not produce,
and `init.py`'s own tests cover a `[misp]` key surviving an init that only sets
the relays. Those tests describe the behaviour this addendum now makes real
rather than hypothetical.
**These prompts are HAND-TESTED by the user, not unit-tested**, like every
other prompt in the tool. Whether a question reads clearly has no assertion.
**A key in the config is a secret at rest.** The file is already mode 0600 and
so are its backups, which is why that was built before any key existed. No
change, named here so it is not mistaken for an oversight.
## Testing
Tested, because each has a right answer:
- a configured destination with an acceptable IOC gets a row; one without an
acceptable IOC does not, and the URL-less case getting no `urlhaus` row is
the one to write first, since it is the rule's whole point
- an unconfigured destination gets no row, however many IOCs would suit it
- each row carries only the types its destination accepts, asserted against a
case holding an IP, a domain and a URL at once, which is the only shape that
can distinguish all three
- MISP's row carries every IOC, including the domain neither vendor takes
- a second `generate()` on the same manifest and config produces the same rows
- the no-destination warning is written and the exit code is still zero
- the config rejects a non-string key and a non-table section, each naming the
file
- a half-filled MISP pair is refused by `report` with the not-configured exit
code, and does NOT stop `parse` from running, which is the half that would
regress silently
Not tested: the `init` prompts, per the rule above.
**No sweep is required for this change.** Both sweeps exist to prove that no
recipient identifier reaches an IOC or a report body, and this addendum
touches neither `parse.py` nor the content of any body. It adds rows to a
manifest from values the user typed into their own config. Stated explicitly
because `AGENTS.md` requires a sweep when `parse.py` changes, and a reader
should be able to see that the question was asked rather than skipped.
## What this leaves for `submit`
Unchanged from the section above, plus one constraint this addendum creates:
- **the `frozen` marker must be written in the same atomic manifest write as
the first `sent` status.** The spec above already required this; this
addendum now depends on it, because `generate()` rebuilds `destinations[]`
wholesale and the freeze is the only thing standing between that rebuild and
a manifest holding receipts.
- the vendor payload shapes, per destination, against each vendor's real
documentation. URLhaus's needs an `auth.abuse.ch` account to read properly.
- whether `report` learns to write those bodies, or `submit` does. The rows
now exist either way, which was the point of building them here.
|