| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
Skipping a destination at init left the previous one in the file. Sections
a run does not render are preserved verbatim, which is what stops an init
setting only the relays from discarding a key set earlier, but a skip is a
deliberate "do not use this" and the two rules collided. Declining MISP
re-enabled the instance from an earlier run, and MISP is the gate for
everything irreversible, so report would have submitted to an instance the
user had just said no to. Both vendors behaved the same way.
Found by hand-testing the prompts, which is why they are hand-tested: no
unit test covers a sequence of runs, and the suite was green throughout.
write() gains `drop`, the sections the user was ASKED about and skipped,
which join the set _preserved_sections already drops. An unasked section is
still preserved, so the non-interactive path and the existing carry-across
tests are unchanged. The prompt only fires when a section BOTH exists and
was skipped, and Enter keeps it: removal is the destructive answer and has
to be typed.
DESTINATION_ANSWER_KEYS is derived from _DESTINATION_FIELDS rather than
written out again, so a destination added later cannot be forgotten here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0176FYdVfpzUq8S9jecqQqL6
|
|
init._quoted() refuses a double quote rather than escaping it, so a name
or organisation containing one raised out of build() after every other
question had been answered, and _cmd_init turned that into an exit code.
An org name is likelier to carry a quote than an API key is, which is the
path Task 8 had just guarded.
Validated at the prompt that asked for it, which is the rule AGENTS.md
records from the hand test that found four instances of the opposite.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0176FYdVfpzUq8S9jecqQqL6
|
|
The MISP pair is asked and validated TOGETHER, because build() now refuses
a half-answered section: a url with no key is configured-and-broken rather
than skipped. Skipping the key discards the url and says so, rather than
trapping a user who has no key to hand in a repeated question.
A double quote is refused at the prompt that asked for it. build() rejects
one with a ValueError, and by then every other answer has been given, so a
pasted key carrying one would have cost the whole run: the shape of defect
the first hand test found four of.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0176FYdVfpzUq8S9jecqQqL6
|
|
init.build() can now write [misp], [abusedb] and [urlhaus], the three
sections report reads, so a user no longer has to hand-edit the config
for them. Follows the [reporter] pattern: a section is emitted only
when something was answered for it, never with an empty api_key.
A half-answered pair (misp_url given, misp_api_key skipped, or vice
versa) is rejected rather than silently written: config.load() would
classify it "incomplete" and report would refuse, but only after the
user believed setup succeeded. Task 8's prompts are the normal route
in, but build() is pure and a caller can hand-build a dict, so this is
enforced here too.
Also fixes FOOTER, which still promised a single [vendors] table; the
spec settled three separate sections and [misp] is no longer "a later
part" since this plan builds it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0176FYdVfpzUq8S9jecqQqL6
|
|
A [misp] section with url but no api_key (or vice versa) is
configured-and-broken, not skipped. report now refuses it before
writing anything, naming the missing key, while parse and contacts
stay unaffected since neither reads config.load()'s incomplete set.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0176FYdVfpzUq8S9jecqQqL6
|
|
report exits zero and writes bodies for a case with only abuse-desk
contacts, since that is a perfectly good report on its own. But a user
who believes MISP is configured and finds no misp row has a typo'd
section name, and MISP gates everything irreversible, so a stderr
warning naming the config path catches that before submit time.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0176FYdVfpzUq8S9jecqQqL6
|
|
generate() gains a `configured` argument and appends vendor_destinations()
after the email bodies are written. It is an ARGUMENT rather than a config
read, the way the identity already is: report.py stays pure and testable
with no files on disk. Defaulting to nothing configured is the safe
direction, a caller that forgets it writes no vendor row rather than
promising a submission nobody set up.
cli.py passes settings.destinations down, so a config carrying a vendor's
keys produces that vendor's row in the case manifest.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0176FYdVfpzUq8S9jecqQqL6
|
|
Adds DESTINATION_KEYS and two Config fields, destinations and
incomplete, so config.load() reports which of misp/abusedb/urlhaus
have every required key filled with a non-empty string. A half-filled
MISP pair is reported as data via incomplete, never raised: parse and
contacts load this same file and read neither key, and report (task 6)
is the one that will refuse it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0176FYdVfpzUq8S9jecqQqL6
|
|
vendor_destinations() turns the DESTINATIONS table plus a case's IOCs
into destination rows, one per configured destination that has at
least one IOC it accepts. Nothing calls it yet; that wiring is a
later task.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0176FYdVfpzUq8S9jecqQqL6
|
|
Names what each report destination accepts (misp: any type, abusedb:
ipv4/ipv6, urlhaus: url), read from each vendor's own API documentation
on 2026-09-10. Nothing consumes it yet.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0176FYdVfpzUq8S9jecqQqL6
|
|
Backlog item 4 asked for a primary source for the machine-readable part.
It exists, and it is IANA's MARF registry rather than x-arf's own docs.
Checked against that registry and RFC 5965 itself on 2026-09-10.
Report-Type is not a registered field. The registry lists 25 names and it
is absent; it came from this spec's worked example rather than a decision.
Removed, because Feedback-Type already says it. Keeping it would have been
harmless, since RFC 5965 section 6 makes ignoring an unknown field a MUST
for the receiver, but that same section requires an extension field be
registered, and a desk should be able to look up every field in a document
this tool sends.
Feedback-Type moves from abuse to fraud. RFC 5965 registers fraud as
"indicates some kind of fraud or phishing activity" and abuse as
"unsolicited email or some other kind of email abuse". This tool reports
phishing, and some desks route fraud separately from bulk spam.
Reported-Uri becomes Reported-URI, the spelling in the ABNF and the
registration. The RFC's own worked example uses the other one, which is
where this came from. Names are case-insensitive so nothing was broken.
A new test holds the registry's 25 names and asserts every emitted field
is one of them, with Source named as the single deliberate exception, so
a new unregistered field cannot arrive unnoticed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LByBnw83xr9YP85nskzkyE
|
|
Refuses without a configured reporter identity rather than filing a report
with no reply address, and turns a frozen or edited case into an error
message rather than a traceback.
The identity rule is EMAIL REQUIRED, NAME AND ORG OPTIONAL, not the plan's
"all three required". Each [reporter] key is individually skippable at init
and config drops a skipped one rather than storing "", so a partial identity
is a normal shape; text_part() already renders whatever subset is present.
Requiring all three would refuse a config init itself writes without
complaint. The address is different in kind because it becomes the From.
That closes backlog item 5: report.build() read identity["email"] directly
and raised a bare KeyError on a name-without-address identity, which is
exactly what skipping one init prompt while answering another produces. The
check lives in the command rather than the builder, since that is what
decides whether the user sees an exit code and a sentence or a traceback.
Two further defects in the plan's code, both fixed here:
- It called config.load() with no argument, so the global --config option
was silently ignored for this subcommand alone. It now resolves the path
the way _cmd_parse does. Reporting against the wrong identity is the
failure that would have caused.
- Its summary printed counts and no path. Nothing sends these bodies yet, so
reviewing them by hand is the entire point of the command, and the case's
bodies directory is now named in the output.
case.py stays the only writer of the manifest: generate() returns it and
case.save() writes it atomically. Bodies are written before that save, so a
failed save leaves bodies the manifest does not record; the next run finds no
recorded hash, treats them as regenerable rather than hand-edited and
overwrites them, and nothing has been sent, so no machinery is warranted.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LByBnw83xr9YP85nskzkyE
|
|
The "Later parts add further sections here" block was emitted by build(),
so preserved sections were appended below it and it became a comment about
nothing, pointing at a table already underneath it.
It is now a FOOTER constant that build() still appends, so a lone build()
reads as a complete file, and write() strips before adding preserved text
and puts back at the end.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LByBnw83xr9YP85nskzkyE
|
|
The previous commit removed this guard after a mutation check said it was
redundant. The check was wrong: it only exercised runs that skipped the
identity from the start, where [reporter] is never written and the footer
always trails a rendered table.
The sequence that breaks is answering the identity and then re-running and
skipping it. [reporter] is then PRESERVED rather than rendered, so the
footer that trailed it in the old file rides across while this run emits
its own, and the file grows a second copy per run.
Covered by a test naming that sequence, and both guards now fail when
removed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LByBnw83xr9YP85nskzkyE
|
|
The carry-across preserved every line before the first [table], but that
text is build()'s own header, not the user's. A second init emitted it
twice and a third three times, with [general] missing from the tail half,
so the file grew a copy per run. It still parsed, which is why the suite
stayed green and only a hand test found it.
Preserving from the first table onward fixes the trailing "Later parts"
block too: it trails whichever table came last, so it is already inside a
skipped section. A separate guard for it passed its own mutation check
and was removed as redundant.
Required prompts now say so. The relay question and the hop picker cannot
be skipped, because with no trust boundary parse refuses and init is the
route out, but they read like the reporter questions that do offer a skip.
The cases prompt says "Enter for <default>" rather than showing the
default in brackets, which was the same ambiguity in the other direction.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LByBnw83xr9YP85nskzkyE
|
|
Each answer is validated at the prompt that asked for it, and a skipped
answer is absent from the file rather than an empty string.
Two defects fixed beyond the plan, both in the carry-across rule. Sections
build() does not produce are preserved verbatim, and that rule was written
when [general] was the only section it produced. Emitting [reporter] as well
made preservation emit it TWICE, and tomllib refuses a duplicate table
outright, so the rewritten file became unreadable and took the preserved
[misp] key with it. Dropping [reporter] unconditionally instead would have
been the opposite defect: an init that skips all three questions would
silently delete an identity set by hand. What is dropped is now read back off
the rendered text, so it is what this run actually wrote rather than what it
might have written, and a section added to the builder later cannot be
forgotten here.
The email check is deliberately not an RFC 5322 validator. What a typo costs
is a report whose reply address bounces, and the answers that produce that
are a name with no @ at all, a spelled-out "at", and a stray space from a
copy-paste. Anything stricter starts rejecting addresses that work.
Skipping the address is allowed but warned about at the prompt: it is the one
field a report cannot be built without, since it becomes the From.
report.build() raises a bare KeyError on that identity today, which the three
prompts made reachable from a config file for the first time; logged as
backlog item 5 rather than fixed here, because where the check belongs is the
report subcommand.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LByBnw83xr9YP85nskzkyE
|
|
The reporter's identity is the one identifier this tool discloses
deliberately, so it comes from config only and parse never supplies it. An
empty value is absent, the same rule the rest of the config follows.
Three departures from the plan, each a defect in its code:
A non-string value is REJECTED, not dropped. The plan filtered on
isinstance(value, str), so name = 42 or email = ["a@b"] silently vanished
and read back as not-configured. This file already learned that lesson from
ipaddress.ip_network(42) returning a valid-looking 0.0.0.42/32: a wrong
value that reads as plausible is worse than an error. Dropping the email
would strip the reply address from every report while the user believed
they were identified, so the typo is reported against the file that holds
it, the way a non-string trusted_relays entry already is.
The value is stored STRIPPED. The plan tested value.strip() for truthiness
but stored the original, so name = " A Reporter " reached the From
display name as "From: A Reporter <...>", verbatim and unquoted.
Only the three keys the spec names are carried across, and text_part no
longer subscripts them. Each key is individually skippable and config drops
a skipped one, so a partial identity is the normal shape, yet text_part
read identity['name'] and identity['org'] directly: a config naming only an
email raised KeyError on a case that had parsed perfectly. The "Reported
by:" line is now joined from the parts present, so a missing org leaves no
stray comma. A wholly unconfigured identity still raises in build(); how to
refuse that belongs to the cli task, not here.
The dataclass field defaults to an empty dict rather than editing every
construction site, and the mutable-dict-in-a-frozen-dataclass is left as
is: report only reads it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LByBnw83xr9YP85nskzkyE
|
|
case.py stays the only writer of the manifest, so generate returns it and
the caller saves. A modified body is backed up with a timestamp before
--force overwrites it.
Bodies are written and read as BYTES, not text. The plan used read_text
and write_text, which fail two ways at once. The locale encoding is not
the utf-8 body_hash pins, so a body hashed differently under a non-UTF-8
LANG; and text mode applies universal newlines, so a report delimited by
CRLF came back with every CRLF collapsed to LF. The second was live on
every platform: regenerating an untouched case raised Modified, because
the body read back never hashed to what was written. For a sent
destination that hash is the record of what was disclosed, so it has to
be the hash of the bytes on disk. Text mode would also translate the bare
LFs inside the machine-readable part on a CRLF platform, corrupting the
field block a desk's parser reads.
The refusal ordering is now pinned by tests rather than left implicit: a
frozen or modified case must leave the bodies directory byte-for-byte as
it was, asserted over the whole directory rather than one file. bodies
is created without parents, so a missing case directory raises rather
than scattering bodies into a tree generate invented.
The backup test pins the filename, not just the count of .orig files.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LByBnw83xr9YP85nskzkyE
|
|
Content hash rather than mtime, because mtime is wrong in both directions.
Any sent destination freezes the whole case with no override: two desks
holding contradictory accounts of one incident is worse than a stale body.
The marker is tested for PRESENCE rather than truthiness. It is write-once
and monotonic, so an empty or malformed value is a half-written record of a
case a desk may already hold, and the safe direction is to refuse; only an
explicit null counts as absent. A non-mapping marker refuses too rather than
raising AttributeError out of a .get, since cli turns Frozen into a message
and a traceback into a bug report.
The hash pins utf-8 rather than leaving the encoding to the locale: it is
compared against a body read back off disk, possibly under a different LANG,
and for a sent destination it is the record of what was disclosed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LByBnw83xr9YP85nskzkyE
|
|
Three parts and no message/rfc822: the original carries every identifier the
first property keeps out, and text/rfc822-headers is the standard's own
answer for a report that cannot include the message.
The third part filters the manifest's headers against the same whitelist
parse.report_headers() applies, rather than trusting it. The structural
argument is that report cannot disclose what it was never given, and a
manifest is a file the user edits: that is exactly a way it can be given a
To. parse still decides; this refuses to publish what it did not decide for.
A header value is attacker-supplied free text the spec keeps deliberately, so
a newline in a Subject forges a header line in a part read entirely as
headers. RFC 2047 turns the break into a fold instead, and a parser unfolds it
back to one header with the value intact. Applied only when a value carries
something unsafe, since encoding every header costs the desk the legibility
this part exists for.
From is built with headerregistry.Address: "Example Consulting, Ltd" through
an f-string parses back as two addresses, the first a bogus addr-spec with no
domain, so a desk's reply reaches nobody.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xj1ayFRSUQ2u7cwb3S4axE
|
|
An RFC 5965 envelope carrying x-arf fields. 5965 reports are about a message
and these are about indicators, so x-arf's Source fills the gap while the
report-type keeps a standards parser working.
Field values are percent-encoded against header injection. This is reachable,
not theoretical: redact.url_valued_parameters() URL-decodes a redirector's
destination to recover it as an indicator, so a body carrying
"?next=http%3A%2F%2Fa.invalid%2Fx%0AFeedback-Type%3A%20not-abuse" produces,
through parse.iocs() on a real message, an IOC whose value holds a literal
newline followed by text shaped like a field. Emitted verbatim it forges a
field in a report carrying the reporter's identity. Encoding rather than
dropping keeps a genuine redirect target; encoding rather than stripping
keeps the indicator recoverable, since a stripped URL is a different URL a
desk would then act on. The escape covers everything str.splitlines() breaks
on, because Python's own email module raises on U+2028 as readily as on LF,
and it runs over UTF-8 bytes and escapes "%" so the reversal is a true
inverse rather than a second injection one step later.
Four corrections to the plan. Source-IP is emitted once, not once per IP:
RFC 5965 gives it "once maximum" and a repeat displaces the primary rather
than adding an address. Source is omitted when there is no typed indicator
rather than emitted empty, since an empty field asserts that the reported
thing is the empty string. Arrival-Date is dropped rather than copied from
the sender's Date header, which is attacker-controlled and means when the
sender claimed to send, not when our MTA received. sha256 and observation
are left to the text part rather than forced into the nearest field.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xj1ayFRSUQ2u7cwb3S4axE
|
|
The marker was an unescaped trailing backslash, and a backslash is legal in
a URL path, so a value ending in one was indistinguishable from a wrap. An
attacker who read this source could append one and make their own indicator
garble itself in the report an abuse desk reads: an adversarial trigger on
attacker-supplied text, not an edge case.
The short case needed no wrapping at all to corrupt. unwrap() ate the
following line regardless, merging an indicator with its own origin
annotation. On Subject it was worse, absorbing the Date beneath it and
making the "Message as declared" block misstate what the message declared,
which is the one thing that block exists to report faithfully.
Every backslash is now doubled before wrapping and halved on the way back,
and unwrap() tells a marker from content by the PARITY of the trailing run.
Doubling only a trailing one would leave "x\\" encoding as "x\" plus a
marker, the same bug one character along.
A second defect surfaced only under a randomised sweep, after the first fix
and a green suite: a break landing BETWEEN the halves of an escaped pair
splits the run whose parity unwrap() counts, so a real marker reads as
content and the tail is silently dropped. It needs a backslash at exactly
the break column, so no hand-written case found it and 454 of 3538 random
ones did. The wrap now backs off a character rather than splitting a pair.
Verified over 9132 adversarial values, including every backslash pattern up
to length 9 and values that are entirely backslashes: 0 round-trip failures,
0 header-block corruptions, no line over 72 columns.
Values are escaped, never rejected or sanitised: the report says what the
message contained.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xj1ayFRSUQ2u7cwb3S4axE
|
|
The ask goes first, only this desk's own indicators appear, and the redaction
note is unconditional: a desk seeing REDACTED with no explanation may read
the report as doctored.
Nothing is truncated. The plan cut three lines with a [:72] slice and left
the indicator list unwrapped, which is the same defect twice: a cut URL is a
WRONG indicator rather than a short one, and a desk acting on the first 72
characters acts on a resource nobody reported. Long values are divided with
an explicit trailing-backslash continuation instead, and unwrap() is the
exact inverse, so the tests assert reassembly rather than mere presence.
That is what rules out a truncation passing as a wrap.
Origins are mapped to English. "header-list_unsubscribe" is a parser's
vocabulary and reads as debug output; an origin the table does not know is
shown as-is, because losing the only line saying where an indicator came
from is worse than showing an ugly token.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xj1ayFRSUQ2u7cwb3S4axE
|
|
Not an error and not an exit code: a case where nothing resolved still
reaches MISP and the vendors. Visible beats absent, so review can see it
without diffing IOC lists.
Two departures from the plan, both about the pair of lists agreeing.
An indicator is unreportable only when NONE of its contacts produced a
mailable address, not when any one of them failed. Contacts fold by host,
so an indicator can sit in a domain contact that resolved and an IP
contact that did not; listing it regardless puts it in the destinations
and in the "no desk found" list at once, and a reviewer acting on the
second hand-reports something already on its way to a desk.
An abuse value with no "@", or with either half empty, no longer becomes
a destination. RDAP jCard data is third-party and occasionally malformed,
and such a value produced a pending destination with an unsendable
target, which the unreportable array then skipped because the contact
did have an abuse entry. Both functions now apply one mailability test,
so the two lists partition the indicators exactly.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xj1ayFRSUQ2u7cwb3S4axE
|
|
"Abuse@Host.Invalid" and "abuse@host.invalid" produced two destinations and
two mails to one desk, which is the duplicate-mail behaviour the grouping
exists to prevent.
The root cause is the test, not the grouping. _group_key folded the domain
correctly, but the test written for it used a lowercase local part in both
of its inputs, so it varied only the half that worked and passed while the
other half was never exercised. A test that varies one half of its input
proves nothing about the other.
The local part now folds for the RFC 2142 role mailboxes. That RFC mandates
those names and requires them matched case-insensitively, so no host runs
"Abuse@" and "abuse@" as different desks and treating them as two costs a
duplicate mail with nothing on the other side of the trade. This reverses
the earlier decision only for the standardised names: any other local part
is still left as published, because for a named mailbox folding could merge
two desks a host genuinely distinguishes, and a dropped desk is worse than a
duplicate. Each half now folds on the strength of its own standard.
Both folds are mutation-checked and killed by distinct tests, so neither
masks the other, and the role fold is checked across all five role names.
Also pins two behaviours that held only by luck: the caller's contacts are
not modified, and a destination's ioc list is not aliased to the contact's.
Both are true today because the grouping starts a fresh list, and nothing
would have caught an implementation that reused the contact's.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xj1ayFRSUQ2u7cwb3S4axE
|
|
A positional id names a slot in one run's list, and report may run again on
a case whose contacts have changed since. A newly resolved indicator whose
desk sorts ahead renumbers every desk after it, so bodies/<id>.xarf on disk
comes to belong to a different desk than the manifest entry sharing that id.
The body hash recorded per destination would then compare one desk's body
against another's, reporting an edit nobody made or missing one that was.
Hashing the address makes an id follow the desk. It hashes the same
normalised form the grouping uses, so two spellings of one desk share an id
rather than letting whichever spelling RDAP published first decide a body's
filename. Eight hex chars is a deliberate ceiling: collision probability is
about 1e-8 at ten desks, and a short id keeps a case directory readable to
the person reviewing it.
Also strengthens two tests that passed against trivially wrong code: the
no-address case now asserts alongside a contact that does resolve, so
returning nothing at all is no longer a passing answer, and the two-address
case asserts the ioc lists and distinct ids rather than the sorted targets
alone.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xj1ayFRSUQ2u7cwb3S4axE
|
|
Two contacts can resolve to the same desk, an IP and a domain at one hoster
being the common case, and grouping per contact would send that desk two
mails about one incident.
The domain half of an address is folded to lowercase before grouping, since
it is case-insensitive by definition; the local part is left verbatim,
because only the receiving host knows whether it folds, and folding two
desks a host distinguishes would silently drop one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xj1ayFRSUQ2u7cwb3S4axE
|
|
The for-clause rule anchored on the clause terminator, which is strictly
stronger than RFC 5321 4.4: For sits inside Opt-info, so With, ID, Via or a
CFWS comment may legitimately follow it, and its ABNF is 1*( Path / Mailbox )
where Mailbox carries no angle brackets. Only the tidiest shape was stripped.
Four ordinary ones published the victim's address in the one header a report
reproduces verbatim, including "for <a@b> (envelope-from <c@d>);", which is
routine Exim and Sendmail output.
Anchored on the address instead. The envelope SENDER survives the cut, since
that is what the report is about, and the leftover whitespace, orphan
separator and emptied comment are tidied because the line is published to a
third party verbatim. Table-driven over all five shapes; reverting the regex
fails six of nine tests in the class.
Two accepted disclosures are now named in the spec's "Where the headers come
from" rather than living in a test comment: our own relay's hostname, which
the by and authserv-id clauses carry by construction, and attacker-controlled
free text in Subject and the From display name, which the whitelist does not
and should not filter. The second is asserted as a documented limit so it
reads as deliberate, and the manifest assertion now bars the obfuscated
you%40example.org spelling alongside the plain one.
case.create() seeds an empty headers block so an unparsed case has the same
shape as a parsed one. Additive, so FORMAT_VERSION is unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xj1ayFRSUQ2u7cwb3S4axE
|
|
report must never open source.eml, so parse decides once what may be
published and report formats only what it is given. To, Cc, Delivered-To and
X-Original-To are absent by construction rather than stripped.
Received is cut to the boundary hop alone, in both directions. Above it are
our own relays; below it is the attacker's own writing, and a forged chain
names an innocent third party there, so publishing a hop below the boundary
puts someone else's address into a report a desk will act on. That is the
third property applied to disclosure rather than to sending_ip().
Truncating the chain was not sufficient on its own: the surviving line is
written by our own relay and records the envelope recipient in its optional
"for <addr>" clause, so the whitelist alone would have published the
victim's address verbatim in the one header a report reproduces in full.
The clause is removed and the rest of the hop kept.
The manifest-wide "no example.org" assertion is narrowed to the headers
block only, where the whitelist deliberately publishes our receiving relay's
name in a by/authserv-id clause. The address itself is still barred there,
asserted separately.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xj1ayFRSUQ2u7cwb3S4axE
|
|
An RDAP query discloses what the user is looking at, and property 1
covers only what is published, so the query rule needed stating in its
own right beside the other three. The section records the three leaks it
came from, because each one was the same shape: validation applied per
branch, forgotten on the next.
Also moves rdap.py's imports into one block at the top. Pure move, the
suite is 192 either side of it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
|
|
6d5822b quoted the interpolated component with safe="", which encoded the
colons of every IPv6 address: 2001:db8::1 went on the wire as
2001%3Adb8%3A%3A1.
RFC 9082 specifies the IPv6 object path segment as the address in its text
form, and RFC 3986 pchar permits ":" in a segment, so the colon must stay
literal. The failure mode was the bad one for this tool: a registry that
does not normalise before matching answers 404, which reads here as "this
netblock publishes no abuse desk" rather than "we asked the wrong
question", losing the abuse contact silently on exactly the IPv6 hops the
tool exists to report.
safe=":" keeps everything the original commit was for. "/" is still
encoded, so the traversal containment is unchanged, and "%" is still
encoded, so a scope id stays contained.
The regression got through because the happy-path guard used an IPv4
address only. The new test asserts a documentation-range IPv6 address
round-trips into the URL unencoded.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
|
|
A re-run overwrites contacts[] wholesale rather than merging. A merge
would let a contact resolved a week ago survive into a report filed
today, which is the stale-address hazard the response caching policy
already refuses, and overwriting makes a re-run always safe, which
matters because a partial network failure makes re-running the natural
next step.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
|
|
contacts.is_queryable() is the admission point and remains the real gate,
but this property has leaked three times already, each time the same shape:
a validator applied to one branch and forgotten on its sibling. First the
url branch was guarded and domain leaked, then domain was guarded and the
IP branch leaked through an IPv6 scope id.
query_ip and query_domain interpolated their component straight into the
path, so a caller reaching them without passing through worklist() could
put a separator, a query or a fragment on the wire. safe="" rather than
urllib's default safe="/" is the point: the default leaves the path
separator intact, which is exactly the traversal shape that leaked before.
query_domain still returns the UNQUOTED candidate, because that is what the
manifest records and the review dialog shows.
The tests call both functions directly, bypassing contacts, since a future
branch that skips the admission point is the failure this layer exists to
survive.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
|
|
Four holes of one class, all in the path that admits a query.
ipaddress.ip_address accepts a scope id since Python 3.9, so
"fe80::1%victim@example.org" passed _is_ip, is_queryable returned on
that branch without inspecting the string further, and rdap.query_ip
interpolated the whole thing into the query URL unquoted. A recipient
identifier reached a registry, which is exactly the disclosure the
fourth property exists to prevent and exactly what the domain-branch
fix closed. Reject a scope id at the admission point. fe80::/10 is
link-local and never a legitimate RDAP target anyway.
The ip branch of worklist() dropped an unparseable value silently
rather than funnelling it through add() like the other two branches,
so a mangled indicator vanished from the manifest instead of showing
up as unusable. It now goes through add(), and is_queryable grew the
check that makes that classification correct: an all-digit last label
is never a TLD, so "999.999.999.999" is a malformed IP rather than a
host to ask a registry about.
Two behaviours had no test and both mutations survived. _host_of's
root-dot strip is now covered through the url branch, since failing to
fold discloses one host to a registry twice; the .lower() half was
dead work because urlsplit already lowercases, and is dropped with a
comment saying so. The two-label requirement is asserted on
is_queryable directly, because a single-label name matches no tld in
the bootstrap and a resolve test could never fail whatever the
validator decides.
The refusal message named no reason, leaving a reviewer unable to tell
whether the parser mangled a legitimate host or the attacker planted
something. Two cases now: a non-ASCII name is plausibly a real IDN
indicator to chase by hand, an illegal character is hostile.
Also let KeyboardInterrupt and SystemExit through resolve()'s handler,
so stopping a long run is not recorded as a failed query on whichever
indicator was in flight.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
|
|
THE FOURTH PROPERTY was breached through the domain branch of worklist().
The url branch is cleaned by _host_of, which uses urlsplit().hostname,
and the domain branch did only value.strip(".").lower(). rdap.query_domain
then interpolates that value into the fetch URL with no quoting.
parse._domain_of takes everything after the @ of a From, Sender or
Reply-To addr-spec, and email.utils.parseaddr permits /, ?, # and %
there, so the whole shape is attacker-controlled through a header they
own. A From of `Bank <phish@victim%40example.org.invalid>` sent the
recipient's own address to a registry, which is precisely the identity
disclosure the property exists to prevent, reaching a third party.
`a/../../x.invalid` escaped the /domain/ endpoint altogether, and query,
fragment and space values all reached the wire.
Fixed at the SINGLE admission point rather than in the offending branch,
because per-branch validation is what failed here: one branch was
cleaned, the next was written without it. is_queryable() now guards
worklist()'s add(), so domain, url and any future branch pass through it.
A rejected value is not dropped. It keeps an entry with an empty abuse
list and an error saying it was never queried, following the rule
suspect_path_segments already sets: flagged and visible to the user
during review, because silent was the bug.
A non-ASCII host is refused rather than encoded to punycode. Guessing the
encoding of an attacker-supplied name is a query that cannot be
justified, and the ceiling is noted in a ponytail comment.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
|
|
abuse_addresses already treats an RDAP response as attacker-controlled.
The bootstrap document comes off the same network and was trusted
completely, which is backwards: it decides WHICH server is asked, so
subverting it is worth more than subverting an answer.
Three ways that hurt the user:
A non-string CIDR was passed to ipaddress.ip_network, which does not
raise on an integer, it returns a /32. A /32 is the longest possible
prefix, so a planted integer won every longest-prefix contest and steered
the query for the attacker's own address to a server they control. That
server names any abuse address it likes and the user files the phishing
report to the phisher. config.py documents this exact trap for the
trusted relays; the same mistake was repeated here.
A malformed entry raised IndexError, TypeError or KeyError straight out
of server_for_ip. bootstrap() writes whatever JSON it receives to the
cache with no schema check, so one bad response is persisted and crashes
every contacts run for seven days with a traceback pointing at nothing
the user can act on. Entry shape is now guarded in the style
abuse_addresses uses and a bad entry is skipped, never fatal.
_NoDowngradeRedirectHandler guards redirects only, so a bootstrap naming
an http:// base sent the initial query in clear text, disclosing which
netblock the user is investigating to anyone on the path. The selected
URL must now be an https string, which makes the guarantee the module
docstring promises hold end to end rather than only on the redirect path.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
|
|
Failure is per query and never stops the run: a timeout on one indicator
must not cost the contacts that did resolve, and a missing contact is a
normal outcome rather than an error.
Bootstraps are passed in rather than fetched here, so this stays
testable with no network and the caller owns the cache policy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
|
|
Adds the fourth non-negotiable property: a query carries a bare host or
IP and never a URL. An RDAP query discloses what the user is looking at,
and a URL path can carry recipient identity that parse deliberately
flags rather than redacts, because a path segment may be the thing being
reported. That decision is safe only while the URL stays local.
Property 1 governs what is published and a query appears in no report,
so property 1 does not cover this and this property does.
Hosts fold, so twenty URLs on one host make one query while the item
keeps every indicator id behind it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
|
|
RDAP wants the registrable domain and a deep host is not one. Rather
than bundling a Public Suffix List, which is a transcribed table that
goes stale weekly and is the failure init.PROVIDERS already documents,
this asks the registry, which is the authority on what is registrable.
The walk is capped and never queries a bare TLD.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
|
|
Only an entity whose roles contain abuse counts. No fallback to a
technical or registrant contact, who is a named human that never
volunteered for abuse mail, and no fallback to abuse@<domain> by
convention: for a phishing domain that mailbox belongs to the attacker,
so constructing it would confirm both the catch and that the reporter's
address is live.
Addresses are validated where they enter rather than where they are
used, because the value becomes a mail recipient later and a control
character in it is header injection into mail this tool sends.
Entity recursion is depth capped so remote JSON cannot hang the tool.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
|
|
Longest prefix rather than first match: a block delegated to a new
operator appears as a more specific range inside its parent, and the
wider range would name the operator that gave it away.
A TLD that publishes no RDAP service selects nothing, which is a normal
outcome for many TLDs rather than a defect.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
|
|
Seven day TTL under XDG_CACHE_HOME, deliberately not in a case
directory: this is a copy of a public map, not evidence.
A failed refetch falls back to the stale copy. Staleness is safe in this
direction, since a server that has moved simply misses and reads as no
contact, while losing IANA entirely would stop the user filing a report.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
|
|
The only socket in this tool. A redirect is remote data directing our
next request, so hops are capped and an https to http downgrade is
refused: a downgraded query travels in clear text and discloses which
netblock is under investigation to anyone on the path.
The timeout is mandatory rather than defaulted, because urllib with no
timeout blocks forever and a hung registry would hang a review.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
|
|
A sweep of the user's real spam found List-Unsubscribe naming a domain
that appeared nowhere else in the message. It is attacker infrastructure
and was going unreported.
Every url from that header goes through redact.url() like a body url: an
unsubscribe link has to say who is unsubscribing, which makes it one of
the likeliest carriers of a recipient token. mailto: entries are skipped
rather than redacted, since the address is the whole value and nothing
useful survives removing it.
Sender is collected on the same terms as Reply-To, included only when it
differs from From. One repeating From is noise; one naming a separate
relay is the infrastructure behind the run.
Also drops the unused urlencode import left in redact.py when
_redact_kv_string stopped using urllib to rebuild the query string.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019NHaqA1Rz5ybed7wFUeQbK
|
|
A display name naming a recognisable address is a deliberate act and a
real signal, so it is reported rather than merely ignored once
_domain_of() stopped mistaking it for the sender.
The IOC carries no value. One of the identities an attacker impersonates
is the recipient themselves, so publishing the impersonated domain would
leak the recipient's own domain in exactly the case worth flagging. What
travels is only that it happened; the reviewer has the message and can
see who was impersonated, and a third party does not need to.
display_name_addresses() still returns the full address for local review.
Only the published IOC list is stripped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019NHaqA1Rz5ybed7wFUeQbK
|
|
_ADDR_DOMAIN.search() returned the first @domain anywhere in the raw
header text. A display name sits before the angle brackets and is
attacker-controlled, so it won.
Two ways that reached a published report. The sender was misattributed:
"Billing at billing@innocent.example" <phish@sender.example.invalid>
filed the report against a third party who sent nothing. And it defeated
the structural guarantee in sender_domains(): the module reads no
recipient header, but an attacker who writes the victim's own address
into the display name hands it one anyway, and it came back out as a
sender domain.
parseaddr() parses the header grammar rather than scanning it, so a
quoted display name cannot supply the address.
leaky.eml's display name now carries the recipient address. The existing
test_no_ioc_holds_a_recipient_address assertion catches this class; it
was green before only because the fixture used a harmless domain.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019NHaqA1Rz5ybed7wFUeQbK
|
|
_extract_ip returned the FIRST bracketed IP in a Received header value.
Postfix (and others) write the client's own HELO/EHLO argument first
and the address it actually observed on the connection second:
Received: from [198.51.100.7] (unknown [203.0.113.99]) by mx...
The first bracket is entirely attacker-chosen; a client can HELO with
any literal it likes. sending_ip() returned 198.51.100.7, reporting
whoever the attacker named rather than 203.0.113.99, the address the
accepting server itself wrote. This needs no forged extra hop, only a
client that HELOs with an address literal, and the module's own
docstring already stated the intended answer ("the bracketed literal
after the connecting hostname") without the code implementing it.
_extract_ip now collects every bracketed, ipaddress-valid literal with
its position and, when there is more than one, prefers the last one
appearing before " by " (the accepting server's own clause, and the
one closest to it). A header with a single bracketed IP or no " by "
token keeps the previous single-candidate behaviour, so simple.eml
(203.0.113.42) and forged-chain.eml (203.0.113.99, item 3's own
mutation-checked test) are unaffected.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
|
|
parts.hostname returns an IPv6 literal WITHOUT its brackets
(2001:db8::1, not [2001:db8::1]), and _netloc_without_userinfo
reassembled f"{host}:{port}" directly from it:
redact.url("http://victim@[2001:db8::1]:8080/p?x=1")
-> "http://2001:db8::1:8080/p?x=REDACTED"
That string is not parseable back into a host and a port, and the
digits after the second-to-last colon are not even part of the address
any more. Reporting it means the abuse desk cannot identify the host
at all, or misreads it, which is the same class of harm as reporting a
wrong IP outright.
_netloc_without_userinfo re-adds brackets whenever the hostname
contains ":", so an IPv6 host now survives userinfo stripping exactly
as an IPv4 or named host already did.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
|
|
parse_qsl reads "?victim@example.org" as the pair
("victim@example.org", ""), and the code kept parameter NAMES because
they fingerprint the phishing kit. A name that is itself an address is
not a name, so keeping it published the recipient's identifier
verbatim (percent-decoded, no less: %40 fools no consumer).
_redact_kv_string now splits the query/fragment string by hand on "&"
and ";" and inspects each token's own name: one containing "@" is a
value that landed in name position and is redacted WHOLE ("?REDACTED"
rather than "?victim%40example.org=REDACTED"); an ordinary name still
keeps its shape ("?flag" stays "?flag=REDACTED", "?t=1&t=2" stays
"?t=REDACTED&t=REDACTED").
This also exposed a second leak reachable through the same fixture:
parse._suspect_segments() ran redact.suspect_path_segments()'s
decode-and-check predicate (meant for a querystring smuggled past
percent-encoding into a PATH segment) against a raw query VALUE, so a
plaintext "?e=you@example.org" reproduced the address in the
manifest's suspect_path_segments flag even though the URL itself was
correctly redacted. redact.suspect_path_segments() now exposes the
opaque-shape half of its check as _looks_opaque(), and parse.py uses
only that half against query values: a query value is always fully
redacted regardless, so the flag may hint at its shape but must never
reproduce it.
Turns tests.test_parse.TestIocAssembly.test_the_address_does_not_survive_any_url_shape
green, and closes the gap test_no_ioc_holds_a_recipient_address had
been passing over with URL redaction fully disabled.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
|
|
Five defects from the user's hand test of the interactive setup, four of
them the same mistake: an answer was validated somewhere other than where
it was given, so a typo cost the whole run.
A bad CIDR was only rejected after the NEXT question had been asked and
answered, and both answers were then discarded. It is validated at its own
prompt now and re-asks, so a mistake costs one retry.
A provider name typed at the first prompt was treated as a CIDR, so
"Gmail" produced "does not appear to be an IPv4 or IPv6 network", which
names nothing the user can act on. The prompt takes either answer now and
tries the provider table first, because the user does not know, and should
not have to know, which of the two questions they are answering.
Picking a hop number out of range raised a bare IndexError, and a
non-numeric pick leaked int()'s own message about base 10. Both re-ask
naming the valid range.
The first question said what to type without saying what it was for. It
now states that the boundary decides which address gets reported, which is
the fact that makes a wrong answer matter.
Ctrl+C and EOF are caught in main() and report that nothing was written.
Setup writes nothing until every answer is in hand, so abandoning it
midway leaves no half-written config, and that was already true; it just
said so with a traceback.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
|
