| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
Backlog item 4 asked for a primary source for the machine-readable part.
It exists, and it is IANA's MARF registry rather than x-arf's own docs.
Checked against that registry and RFC 5965 itself on 2026-09-10.
Report-Type is not a registered field. The registry lists 25 names and it
is absent; it came from this spec's worked example rather than a decision.
Removed, because Feedback-Type already says it. Keeping it would have been
harmless, since RFC 5965 section 6 makes ignoring an unknown field a MUST
for the receiver, but that same section requires an extension field be
registered, and a desk should be able to look up every field in a document
this tool sends.
Feedback-Type moves from abuse to fraud. RFC 5965 registers fraud as
"indicates some kind of fraud or phishing activity" and abuse as
"unsolicited email or some other kind of email abuse". This tool reports
phishing, and some desks route fraud separately from bulk spam.
Reported-Uri becomes Reported-URI, the spelling in the ABNF and the
registration. The RFC's own worked example uses the other one, which is
where this came from. Names are case-insensitive so nothing was broken.
A new test holds the registry's 25 names and asserts every emitted field
is one of them, with Source named as the single deliberate exception, so
a new unregistered field cannot arrive unnoticed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LByBnw83xr9YP85nskzkyE
|
|
Sweep A over 92 real messages after task 1 changed parse.py: 1685
indicators, 92 bodies, 0 crashes, 0 empty parses, no address from a raw
source in the IOC output and no recipient address in any generated body.
What it measured is the limit the spec already accepts. Subject is
published verbatim, and 14 of the 92 messages carried the recipient's
local part inside it because the kit personalises the lure. None carried
it in the From display name, and the envelope recipient was cut from the
boundary Received line in every message. That is the whitelist governing
which headers travel rather than what is inside one, which the spec's
"attacker-controlled free text is published unfiltered" section states
outright and names the sweep as the cover for.
personalised-subject.eml pins all three behaviours, the accepted one
included, so the number cannot drift unnoticed. The for-clause test is
mutation-checked: stop cutting the clause and all three fail.
Sweep B, 12 hand-picked public targets and none from the corpus: 12 of 12,
0 failures, all five RIRs parseable, IPv6 live, the label walk and the
multi-part suffix both correct.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LByBnw83xr9YP85nskzkyE
|
|
The reporter's identity is the one identifier this tool discloses
deliberately, so it comes from config only and parse never supplies it. An
empty value is absent, the same rule the rest of the config follows.
Three departures from the plan, each a defect in its code:
A non-string value is REJECTED, not dropped. The plan filtered on
isinstance(value, str), so name = 42 or email = ["a@b"] silently vanished
and read back as not-configured. This file already learned that lesson from
ipaddress.ip_network(42) returning a valid-looking 0.0.0.42/32: a wrong
value that reads as plausible is worse than an error. Dropping the email
would strip the reply address from every report while the user believed
they were identified, so the typo is reported against the file that holds
it, the way a non-string trusted_relays entry already is.
The value is stored STRIPPED. The plan tested value.strip() for truthiness
but stored the original, so name = " A Reporter " reached the From
display name as "From: A Reporter <...>", verbatim and unquoted.
Only the three keys the spec names are carried across, and text_part no
longer subscripts them. Each key is individually skippable and config drops
a skipped one, so a partial identity is the normal shape, yet text_part
read identity['name'] and identity['org'] directly: a config naming only an
email raised KeyError on a case that had parsed perfectly. The "Reported
by:" line is now joined from the parts present, so a missing org leaves no
stray comma. A wholly unconfigured identity still raises in build(); how to
refuse that belongs to the cli task, not here.
The dataclass field defaults to an empty dict rather than editing every
construction site, and the mutable-dict-in-a-frozen-dataclass is left as
is: report only reads it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LByBnw83xr9YP85nskzkyE
|
|
case.py stays the only writer of the manifest, so generate returns it and
the caller saves. A modified body is backed up with a timestamp before
--force overwrites it.
Bodies are written and read as BYTES, not text. The plan used read_text
and write_text, which fail two ways at once. The locale encoding is not
the utf-8 body_hash pins, so a body hashed differently under a non-UTF-8
LANG; and text mode applies universal newlines, so a report delimited by
CRLF came back with every CRLF collapsed to LF. The second was live on
every platform: regenerating an untouched case raised Modified, because
the body read back never hashed to what was written. For a sent
destination that hash is the record of what was disclosed, so it has to
be the hash of the bytes on disk. Text mode would also translate the bare
LFs inside the machine-readable part on a CRLF platform, corrupting the
field block a desk's parser reads.
The refusal ordering is now pinned by tests rather than left implicit: a
frozen or modified case must leave the bodies directory byte-for-byte as
it was, asserted over the whole directory rather than one file. bodies
is created without parents, so a missing case directory raises rather
than scattering bodies into a tree generate invented.
The backup test pins the filename, not just the count of .orig files.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LByBnw83xr9YP85nskzkyE
|
|
Content hash rather than mtime, because mtime is wrong in both directions.
Any sent destination freezes the whole case with no override: two desks
holding contradictory accounts of one incident is worse than a stale body.
The marker is tested for PRESENCE rather than truthiness. It is write-once
and monotonic, so an empty or malformed value is a half-written record of a
case a desk may already hold, and the safe direction is to refuse; only an
explicit null counts as absent. A non-mapping marker refuses too rather than
raising AttributeError out of a .get, since cli turns Frozen into a message
and a traceback into a bug report.
The hash pins utf-8 rather than leaving the encoding to the locale: it is
compared against a body read back off disk, possibly under a different LANG,
and for a sent destination it is the record of what was disclosed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LByBnw83xr9YP85nskzkyE
|
|
Three parts and no message/rfc822: the original carries every identifier the
first property keeps out, and text/rfc822-headers is the standard's own
answer for a report that cannot include the message.
The third part filters the manifest's headers against the same whitelist
parse.report_headers() applies, rather than trusting it. The structural
argument is that report cannot disclose what it was never given, and a
manifest is a file the user edits: that is exactly a way it can be given a
To. parse still decides; this refuses to publish what it did not decide for.
A header value is attacker-supplied free text the spec keeps deliberately, so
a newline in a Subject forges a header line in a part read entirely as
headers. RFC 2047 turns the break into a fold instead, and a parser unfolds it
back to one header with the value intact. Applied only when a value carries
something unsafe, since encoding every header costs the desk the legibility
this part exists for.
From is built with headerregistry.Address: "Example Consulting, Ltd" through
an f-string parses back as two addresses, the first a bogus addr-spec with no
domain, so a desk's reply reaches nobody.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xj1ayFRSUQ2u7cwb3S4axE
|
|
An RFC 5965 envelope carrying x-arf fields. 5965 reports are about a message
and these are about indicators, so x-arf's Source fills the gap while the
report-type keeps a standards parser working.
Field values are percent-encoded against header injection. This is reachable,
not theoretical: redact.url_valued_parameters() URL-decodes a redirector's
destination to recover it as an indicator, so a body carrying
"?next=http%3A%2F%2Fa.invalid%2Fx%0AFeedback-Type%3A%20not-abuse" produces,
through parse.iocs() on a real message, an IOC whose value holds a literal
newline followed by text shaped like a field. Emitted verbatim it forges a
field in a report carrying the reporter's identity. Encoding rather than
dropping keeps a genuine redirect target; encoding rather than stripping
keeps the indicator recoverable, since a stripped URL is a different URL a
desk would then act on. The escape covers everything str.splitlines() breaks
on, because Python's own email module raises on U+2028 as readily as on LF,
and it runs over UTF-8 bytes and escapes "%" so the reversal is a true
inverse rather than a second injection one step later.
Four corrections to the plan. Source-IP is emitted once, not once per IP:
RFC 5965 gives it "once maximum" and a repeat displaces the primary rather
than adding an address. Source is omitted when there is no typed indicator
rather than emitted empty, since an empty field asserts that the reported
thing is the empty string. Arrival-Date is dropped rather than copied from
the sender's Date header, which is attacker-controlled and means when the
sender claimed to send, not when our MTA received. sha256 and observation
are left to the text part rather than forced into the nearest field.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xj1ayFRSUQ2u7cwb3S4axE
|
|
The marker was an unescaped trailing backslash, and a backslash is legal in
a URL path, so a value ending in one was indistinguishable from a wrap. An
attacker who read this source could append one and make their own indicator
garble itself in the report an abuse desk reads: an adversarial trigger on
attacker-supplied text, not an edge case.
The short case needed no wrapping at all to corrupt. unwrap() ate the
following line regardless, merging an indicator with its own origin
annotation. On Subject it was worse, absorbing the Date beneath it and
making the "Message as declared" block misstate what the message declared,
which is the one thing that block exists to report faithfully.
Every backslash is now doubled before wrapping and halved on the way back,
and unwrap() tells a marker from content by the PARITY of the trailing run.
Doubling only a trailing one would leave "x\\" encoding as "x\" plus a
marker, the same bug one character along.
A second defect surfaced only under a randomised sweep, after the first fix
and a green suite: a break landing BETWEEN the halves of an escaped pair
splits the run whose parity unwrap() counts, so a real marker reads as
content and the tail is silently dropped. It needs a backslash at exactly
the break column, so no hand-written case found it and 454 of 3538 random
ones did. The wrap now backs off a character rather than splitting a pair.
Verified over 9132 adversarial values, including every backslash pattern up
to length 9 and values that are entirely backslashes: 0 round-trip failures,
0 header-block corruptions, no line over 72 columns.
Values are escaped, never rejected or sanitised: the report says what the
message contained.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xj1ayFRSUQ2u7cwb3S4axE
|
|
The ask goes first, only this desk's own indicators appear, and the redaction
note is unconditional: a desk seeing REDACTED with no explanation may read
the report as doctored.
Nothing is truncated. The plan cut three lines with a [:72] slice and left
the indicator list unwrapped, which is the same defect twice: a cut URL is a
WRONG indicator rather than a short one, and a desk acting on the first 72
characters acts on a resource nobody reported. Long values are divided with
an explicit trailing-backslash continuation instead, and unwrap() is the
exact inverse, so the tests assert reassembly rather than mere presence.
That is what rules out a truncation passing as a wrap.
Origins are mapped to English. "header-list_unsubscribe" is a parser's
vocabulary and reads as debug output; an origin the table does not know is
shown as-is, because losing the only line saying where an indicator came
from is worse than showing an ugly token.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xj1ayFRSUQ2u7cwb3S4axE
|
|
Not an error and not an exit code: a case where nothing resolved still
reaches MISP and the vendors. Visible beats absent, so review can see it
without diffing IOC lists.
Two departures from the plan, both about the pair of lists agreeing.
An indicator is unreportable only when NONE of its contacts produced a
mailable address, not when any one of them failed. Contacts fold by host,
so an indicator can sit in a domain contact that resolved and an IP
contact that did not; listing it regardless puts it in the destinations
and in the "no desk found" list at once, and a reviewer acting on the
second hand-reports something already on its way to a desk.
An abuse value with no "@", or with either half empty, no longer becomes
a destination. RDAP jCard data is third-party and occasionally malformed,
and such a value produced a pending destination with an unsendable
target, which the unreportable array then skipped because the contact
did have an abuse entry. Both functions now apply one mailability test,
so the two lists partition the indicators exactly.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xj1ayFRSUQ2u7cwb3S4axE
|
|
"Abuse@Host.Invalid" and "abuse@host.invalid" produced two destinations and
two mails to one desk, which is the duplicate-mail behaviour the grouping
exists to prevent.
The root cause is the test, not the grouping. _group_key folded the domain
correctly, but the test written for it used a lowercase local part in both
of its inputs, so it varied only the half that worked and passed while the
other half was never exercised. A test that varies one half of its input
proves nothing about the other.
The local part now folds for the RFC 2142 role mailboxes. That RFC mandates
those names and requires them matched case-insensitively, so no host runs
"Abuse@" and "abuse@" as different desks and treating them as two costs a
duplicate mail with nothing on the other side of the trade. This reverses
the earlier decision only for the standardised names: any other local part
is still left as published, because for a named mailbox folding could merge
two desks a host genuinely distinguishes, and a dropped desk is worse than a
duplicate. Each half now folds on the strength of its own standard.
Both folds are mutation-checked and killed by distinct tests, so neither
masks the other, and the role fold is checked across all five role names.
Also pins two behaviours that held only by luck: the caller's contacts are
not modified, and a destination's ioc list is not aliased to the contact's.
Both are true today because the grouping starts a fresh list, and nothing
would have caught an implementation that reused the contact's.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xj1ayFRSUQ2u7cwb3S4axE
|
|
A positional id names a slot in one run's list, and report may run again on
a case whose contacts have changed since. A newly resolved indicator whose
desk sorts ahead renumbers every desk after it, so bodies/<id>.xarf on disk
comes to belong to a different desk than the manifest entry sharing that id.
The body hash recorded per destination would then compare one desk's body
against another's, reporting an edit nobody made or missing one that was.
Hashing the address makes an id follow the desk. It hashes the same
normalised form the grouping uses, so two spellings of one desk share an id
rather than letting whichever spelling RDAP published first decide a body's
filename. Eight hex chars is a deliberate ceiling: collision probability is
about 1e-8 at ten desks, and a short id keeps a case directory readable to
the person reviewing it.
Also strengthens two tests that passed against trivially wrong code: the
no-address case now asserts alongside a contact that does resolve, so
returning nothing at all is no longer a passing answer, and the two-address
case asserts the ioc lists and distinct ids rather than the sorted targets
alone.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xj1ayFRSUQ2u7cwb3S4axE
|
|
Two contacts can resolve to the same desk, an IP and a domain at one hoster
being the common case, and grouping per contact would send that desk two
mails about one incident.
The domain half of an address is folded to lowercase before grouping, since
it is case-insensitive by definition; the local part is left verbatim,
because only the receiving host knows whether it folds, and folding two
desks a host distinguishes would silently drop one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xj1ayFRSUQ2u7cwb3S4axE
|
