| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
The test asserted only that resolve() returned one entry. It returns one
entry whether the injected fetch answered or the real transport was
blocked, because resolve() records a transport failure as a per-entry
error rather than raising. So the assertion would have stayed green if
someone removed the injection, which is the one thing the test exists to
notice.
It now asserts the handle that only the injected fetch can produce.
Removing the injection fails with KeyError: 'handle'.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
|
|
The umbrella design verifies property 2 by running with sockets raising.
Now that a network module exists, that has to stay true for the WHOLE
suite rather than for everything except contacts: a network module that
can only be tested with a network is a module that stops being tested.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
|
|
6d5822b quoted the interpolated component with safe="", which encoded the
colons of every IPv6 address: 2001:db8::1 went on the wire as
2001%3Adb8%3A%3A1.
RFC 9082 specifies the IPv6 object path segment as the address in its text
form, and RFC 3986 pchar permits ":" in a segment, so the colon must stay
literal. The failure mode was the bad one for this tool: a registry that
does not normalise before matching answers 404, which reads here as "this
netblock publishes no abuse desk" rather than "we asked the wrong
question", losing the abuse contact silently on exactly the IPv6 hops the
tool exists to report.
safe=":" keeps everything the original commit was for. "/" is still
encoded, so the traversal containment is unchanged, and "%" is still
encoded, so a scope id stays contained.
The regression got through because the happy-path guard used an IPv4
address only. The new test asserts a documentation-range IPv6 address
round-trips into the URL unencoded.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
|
|
A re-run overwrites contacts[] wholesale rather than merging. A merge
would let a contact resolved a week ago survive into a report filed
today, which is the stale-address hazard the response caching policy
already refuses, and overwriting makes a re-run always safe, which
matters because a partial network failure makes re-running the natural
next step.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
|
|
contacts.is_queryable() is the admission point and remains the real gate,
but this property has leaked three times already, each time the same shape:
a validator applied to one branch and forgotten on its sibling. First the
url branch was guarded and domain leaked, then domain was guarded and the
IP branch leaked through an IPv6 scope id.
query_ip and query_domain interpolated their component straight into the
path, so a caller reaching them without passing through worklist() could
put a separator, a query or a fragment on the wire. safe="" rather than
urllib's default safe="/" is the point: the default leaves the path
separator intact, which is exactly the traversal shape that leaked before.
query_domain still returns the UNQUOTED candidate, because that is what the
manifest records and the review dialog shows.
The tests call both functions directly, bypassing contacts, since a future
branch that skips the admission point is the failure this layer exists to
survive.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
|
|
Four holes of one class, all in the path that admits a query.
ipaddress.ip_address accepts a scope id since Python 3.9, so
"fe80::1%victim@example.org" passed _is_ip, is_queryable returned on
that branch without inspecting the string further, and rdap.query_ip
interpolated the whole thing into the query URL unquoted. A recipient
identifier reached a registry, which is exactly the disclosure the
fourth property exists to prevent and exactly what the domain-branch
fix closed. Reject a scope id at the admission point. fe80::/10 is
link-local and never a legitimate RDAP target anyway.
The ip branch of worklist() dropped an unparseable value silently
rather than funnelling it through add() like the other two branches,
so a mangled indicator vanished from the manifest instead of showing
up as unusable. It now goes through add(), and is_queryable grew the
check that makes that classification correct: an all-digit last label
is never a TLD, so "999.999.999.999" is a malformed IP rather than a
host to ask a registry about.
Two behaviours had no test and both mutations survived. _host_of's
root-dot strip is now covered through the url branch, since failing to
fold discloses one host to a registry twice; the .lower() half was
dead work because urlsplit already lowercases, and is dropped with a
comment saying so. The two-label requirement is asserted on
is_queryable directly, because a single-label name matches no tld in
the bootstrap and a resolve test could never fail whatever the
validator decides.
The refusal message named no reason, leaving a reviewer unable to tell
whether the parser mangled a legitimate host or the attacker planted
something. Two cases now: a non-ASCII name is plausibly a real IDN
indicator to chase by hand, an illegal character is hostile.
Also let KeyboardInterrupt and SystemExit through resolve()'s handler,
so stopping a long run is not recorded as a failed query on whichever
indicator was in flight.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
|
|
THE FOURTH PROPERTY was breached through the domain branch of worklist().
The url branch is cleaned by _host_of, which uses urlsplit().hostname,
and the domain branch did only value.strip(".").lower(). rdap.query_domain
then interpolates that value into the fetch URL with no quoting.
parse._domain_of takes everything after the @ of a From, Sender or
Reply-To addr-spec, and email.utils.parseaddr permits /, ?, # and %
there, so the whole shape is attacker-controlled through a header they
own. A From of `Bank <phish@victim%40example.org.invalid>` sent the
recipient's own address to a registry, which is precisely the identity
disclosure the property exists to prevent, reaching a third party.
`a/../../x.invalid` escaped the /domain/ endpoint altogether, and query,
fragment and space values all reached the wire.
Fixed at the SINGLE admission point rather than in the offending branch,
because per-branch validation is what failed here: one branch was
cleaned, the next was written without it. is_queryable() now guards
worklist()'s add(), so domain, url and any future branch pass through it.
A rejected value is not dropped. It keeps an entry with an empty abuse
list and an error saying it was never queried, following the rule
suspect_path_segments already sets: flagged and visible to the user
during review, because silent was the bug.
A non-ASCII host is refused rather than encoded to punycode. Guessing the
encoding of an attacker-supplied name is a query that cannot be
justified, and the ceiling is noted in a ponytail comment.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
|
|
test_the_walk_is_capped could not fail. Its host had nine labels, so the
bare-TLD guard alone bounded the walk to eight attempts and the cap was
never reached, and the assertion read rdap._MAX_LABEL_WALK at assert
time, so the expected value moved with any mutation. Raising the cap to a
billion left the suite green. It now uses a twelve-label host and a
hard-coded expected count of five, and raising the cap fails it with
11 != 5.
test_a_newline_in_an_address_is_rejected asserted a guarantee that a
different function happened to provide. email.utils.parseaddr rejects the
CRLF fixture by itself, returning an empty address, so deleting both
control-character checks from _valid_address left the suite green while
the real defence was gone. parseaddr passes \x0b and \x0c straight
through, and those reach a mail header raw, so a case for each is added
alongside the CRLF one, which is still worth asserting.
The new HostileBootstrap cases cover the bootstrap defects fixed in the
previous commit: a planted integer CIDR, a malformed services value in
seven shapes, a plaintext base URL, and the two paths where a bad entry
must not hide a good one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
|
|
abuse_addresses already treats an RDAP response as attacker-controlled.
The bootstrap document comes off the same network and was trusted
completely, which is backwards: it decides WHICH server is asked, so
subverting it is worth more than subverting an answer.
Three ways that hurt the user:
A non-string CIDR was passed to ipaddress.ip_network, which does not
raise on an integer, it returns a /32. A /32 is the longest possible
prefix, so a planted integer won every longest-prefix contest and steered
the query for the attacker's own address to a server they control. That
server names any abuse address it likes and the user files the phishing
report to the phisher. config.py documents this exact trap for the
trusted relays; the same mistake was repeated here.
A malformed entry raised IndexError, TypeError or KeyError straight out
of server_for_ip. bootstrap() writes whatever JSON it receives to the
cache with no schema check, so one bad response is persisted and crashes
every contacts run for seven days with a traceback pointing at nothing
the user can act on. Entry shape is now guarded in the style
abuse_addresses uses and a bad entry is skipped, never fatal.
_NoDowngradeRedirectHandler guards redirects only, so a bootstrap naming
an http:// base sent the initial query in clear text, disclosing which
netblock the user is investigating to anyone on the path. The selected
URL must now be an https string, which makes the guarantee the module
docstring promises hold end to end rather than only on the redirect path.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
|
|
Failure is per query and never stops the run: a timeout on one indicator
must not cost the contacts that did resolve, and a missing contact is a
normal outcome rather than an error.
Bootstraps are passed in rather than fetched here, so this stays
testable with no network and the caller owns the cache policy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
|
|
Adds the fourth non-negotiable property: a query carries a bare host or
IP and never a URL. An RDAP query discloses what the user is looking at,
and a URL path can carry recipient identity that parse deliberately
flags rather than redacts, because a path segment may be the thing being
reported. That decision is safe only while the URL stays local.
Property 1 governs what is published and a query appears in no report,
so property 1 does not cover this and this property does.
Hosts fold, so twenty URLs on one host make one query while the item
keeps every indicator id behind it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
|
|
The spec named a strict Accept: application/rdap+json while the plan, and
so the code, sends application/rdap+json with application/json at q=0.9.
A spec review caught the disagreement. The wider header is the right
behaviour, since some registries serve plain application/json and a
strict header would lose a contact that was published, so the spec moves
to match the code and now records why.
Both documents also claimed requirements.txt stays empty. There is no
requirements.txt in this repository and there never has been, so the
claim was true only vacuously. They now say the tool declares no
dependencies, which is the thing actually being promised.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
|
|
Two corrections found while executing it.
The offline proof in task 9 patched socket.socket itself, which breaks
the ssl module at import time and produces false failures unrelated to
network use. It now patches socket.socket.connect, which is the right
target.
Task 12 gains a step moving the imports in rdap.py to the top of the
file. The module was built one task at a time, so several imports landed
mid-file after function definitions. Doing it in the documentation pass
keeps it out of the diff that contacts.py builds on.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
|
|
RDAP wants the registrable domain and a deep host is not one. Rather
than bundling a Public Suffix List, which is a transcribed table that
goes stale weekly and is the failure init.PROVIDERS already documents,
this asks the registry, which is the authority on what is registrable.
The walk is capped and never queries a bare TLD.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
|
|
Only an entity whose roles contain abuse counts. No fallback to a
technical or registrant contact, who is a named human that never
volunteered for abuse mail, and no fallback to abuse@<domain> by
convention: for a phishing domain that mailbox belongs to the attacker,
so constructing it would confirm both the catch and that the reporter's
address is live.
Addresses are validated where they enter rather than where they are
used, because the value becomes a mail recipient later and a control
character in it is header injection into mail this tool sends.
Entity recursion is depth capped so remote JSON cannot hang the tool.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
|
|
Longest prefix rather than first match: a block delegated to a new
operator appears as a more specific range inside its parent, and the
wider range would name the operator that gave it away.
A TLD that publishes no RDAP service selects nothing, which is a normal
outcome for many TLDs rather than a defect.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
|
|
Seven day TTL under XDG_CACHE_HOME, deliberately not in a case
directory: this is a copy of a public map, not evidence.
A failed refetch falls back to the stale copy. Staleness is safe in this
direction, since a server that has moved simply misses and reads as no
contact, while losing IANA entirely would stop the user filing a report.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
|
|
The only socket in this tool. A redirect is remote data directing our
next request, so hops are capped and an https to http downgrade is
refused: a downgraded query travels in clear text and discloses which
netblock is under investigation to anyone on the path.
The timeout is mandatory rather than defaulted, because urllib with no
timeout blocks forever and a hung registry would hang a review.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
|
|
Twelve tasks, TDD throughout, building bottom up: transport, bootstrap
cache, server selection, jCard extraction, querying with the label walk,
the worklist, resolution, the subcommand, an offline proof, two sweeps
and the documentation.
The code in the plan was assembled and run before committing it, against
the plan's own tests, 44 of which pass. That found two defects in the
plan itself: a parameter that shadowed the cache_dir function and forced
a globals() lookup, now renamed to cache_root, and a hand-rolled fake
request in the redirect test that lacked the attributes
HTTPRedirectHandler reads, now a real urllib Request.
Also drops the em dashes from the Documents list in AGENTS.md, which the
no-em-dash writing rule covers and which the contacts entry had matched.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
|
|
Settles what the umbrella design left open for this part: RDAP bootstrap,
caching policy, and what happens for a netblock that publishes no abuse
contact.
Scope is IPs and domains, with a URL contributing only its host. The
bootstrap files are fetched from IANA and cached with a long TTL, while
responses are cached in memory for one run only: a stale bootstrap merely
misses, but a stale abuse address sends a report into a dead mailbox
silently. The registrable domain is found by walking up the labels and
asking the registry rather than by bundling a Public Suffix List, which
would be a transcribed table that goes stale weekly.
Address extraction is a strict abuse-role match with no fallback. In
particular it never constructs abuse@<domain>, because for a phishing
domain that mailbox belongs to the attacker and mailing it would confirm
both the catch and that the address is live.
Adds a fourth non-negotiable property: a query carries a bare host or IP
and never a URL. Property 1 governs what is published, and a query is a
disclosure that appears in no report, so a path segment that parse
deliberately flags rather than redacts would otherwise leak.
Transport is urllib with an injected fetch, so requirements.txt stays
empty and the suite keeps passing with sockets raising, for the whole
suite rather than for everything except contacts.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
|
|
A synthetic fixture only ever holds the shapes someone thought of, and
the shapes nobody thought of are the ones that leak. Sweeping the user's
own tag:spam is the only source of those, and it earned its place: the
first message found a header the parser did not read, and the corpus
exercised a spoofed Reply-To display name no fixture had.
AGENTS.md records it under Testing, beside the fixture rule it is the
deliberate exception to: ask first, work from the scratchpad, compare
every address in the raw source against the whole ioc output, and let
only counts and stripped domains out. A finding becomes a synthetic
fixture; the real message never enters the repository.
BACKLOG.md opens with the one thing the sweep found that was not worth
fixing blind: boilerplate namespace urls (w3.org doctypes) reported as
indicators. Noise rather than a defect, and the argument for leaving it
alone is real, so it is written down with both sides rather than built.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019NHaqA1Rz5ybed7wFUeQbK
|
|
A sweep of the user's real spam found List-Unsubscribe naming a domain
that appeared nowhere else in the message. It is attacker infrastructure
and was going unreported.
Every url from that header goes through redact.url() like a body url: an
unsubscribe link has to say who is unsubscribing, which makes it one of
the likeliest carriers of a recipient token. mailto: entries are skipped
rather than redacted, since the address is the whole value and nothing
useful survives removing it.
Sender is collected on the same terms as Reply-To, included only when it
differs from From. One repeating From is noise; one naming a separate
relay is the infrastructure behind the run.
Also drops the unused urlencode import left in redact.py when
_redact_kv_string stopped using urllib to rebuild the query string.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019NHaqA1Rz5ybed7wFUeQbK
|
|
A display name naming a recognisable address is a deliberate act and a
real signal, so it is reported rather than merely ignored once
_domain_of() stopped mistaking it for the sender.
The IOC carries no value. One of the identities an attacker impersonates
is the recipient themselves, so publishing the impersonated domain would
leak the recipient's own domain in exactly the case worth flagging. What
travels is only that it happened; the reviewer has the message and can
see who was impersonated, and a third party does not need to.
display_name_addresses() still returns the full address for local review.
Only the published IOC list is stripped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019NHaqA1Rz5ybed7wFUeQbK
|
|
_ADDR_DOMAIN.search() returned the first @domain anywhere in the raw
header text. A display name sits before the angle brackets and is
attacker-controlled, so it won.
Two ways that reached a published report. The sender was misattributed:
"Billing at billing@innocent.example" <phish@sender.example.invalid>
filed the report against a third party who sent nothing. And it defeated
the structural guarantee in sender_domains(): the module reads no
recipient header, but an attacker who writes the victim's own address
into the display name hands it one anyway, and it came back out as a
sender domain.
parseaddr() parses the header grammar rather than scanning it, so a
quoted display name cannot supply the address.
leaky.eml's display name now carries the recipient address. The existing
test_no_ioc_holds_a_recipient_address assertion catches this class; it
was green before only because the fixture used a harmless domain.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019NHaqA1Rz5ybed7wFUeQbK
|
|
_extract_ip returned the FIRST bracketed IP in a Received header value.
Postfix (and others) write the client's own HELO/EHLO argument first
and the address it actually observed on the connection second:
Received: from [198.51.100.7] (unknown [203.0.113.99]) by mx...
The first bracket is entirely attacker-chosen; a client can HELO with
any literal it likes. sending_ip() returned 198.51.100.7, reporting
whoever the attacker named rather than 203.0.113.99, the address the
accepting server itself wrote. This needs no forged extra hop, only a
client that HELOs with an address literal, and the module's own
docstring already stated the intended answer ("the bracketed literal
after the connecting hostname") without the code implementing it.
_extract_ip now collects every bracketed, ipaddress-valid literal with
its position and, when there is more than one, prefers the last one
appearing before " by " (the accepting server's own clause, and the
one closest to it). A header with a single bracketed IP or no " by "
token keeps the previous single-candidate behaviour, so simple.eml
(203.0.113.42) and forged-chain.eml (203.0.113.99, item 3's own
mutation-checked test) are unaffected.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
|
|
parts.hostname returns an IPv6 literal WITHOUT its brackets
(2001:db8::1, not [2001:db8::1]), and _netloc_without_userinfo
reassembled f"{host}:{port}" directly from it:
redact.url("http://victim@[2001:db8::1]:8080/p?x=1")
-> "http://2001:db8::1:8080/p?x=REDACTED"
That string is not parseable back into a host and a port, and the
digits after the second-to-last colon are not even part of the address
any more. Reporting it means the abuse desk cannot identify the host
at all, or misreads it, which is the same class of harm as reporting a
wrong IP outright.
_netloc_without_userinfo re-adds brackets whenever the hostname
contains ":", so an IPv6 host now survives userinfo stripping exactly
as an IPv4 or named host already did.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
|
|
parse_qsl reads "?victim@example.org" as the pair
("victim@example.org", ""), and the code kept parameter NAMES because
they fingerprint the phishing kit. A name that is itself an address is
not a name, so keeping it published the recipient's identifier
verbatim (percent-decoded, no less: %40 fools no consumer).
_redact_kv_string now splits the query/fragment string by hand on "&"
and ";" and inspects each token's own name: one containing "@" is a
value that landed in name position and is redacted WHOLE ("?REDACTED"
rather than "?victim%40example.org=REDACTED"); an ordinary name still
keeps its shape ("?flag" stays "?flag=REDACTED", "?t=1&t=2" stays
"?t=REDACTED&t=REDACTED").
This also exposed a second leak reachable through the same fixture:
parse._suspect_segments() ran redact.suspect_path_segments()'s
decode-and-check predicate (meant for a querystring smuggled past
percent-encoding into a PATH segment) against a raw query VALUE, so a
plaintext "?e=you@example.org" reproduced the address in the
manifest's suspect_path_segments flag even though the URL itself was
correctly redacted. redact.suspect_path_segments() now exposes the
opaque-shape half of its check as _looks_opaque(), and parse.py uses
only that half against query values: a query value is always fully
redacted regardless, so the flag may hint at its shape but must never
reproduce it.
Turns tests.test_parse.TestIocAssembly.test_the_address_does_not_survive_any_url_shape
green, and closes the gap test_no_ioc_holds_a_recipient_address had
been passing over with URL redaction fully disabled.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
|
|
test_no_ioc_holds_a_recipient_address grepped for the literal
"example.org", and every existing fixture hides the recipient address
as base64, so URL redaction could be disabled entirely and both this
test and test_cli's counterpart stayed green.
leaky.eml carries you@example.org in five URL shapes plus a From
display-name trap. Adding it to the fixture list, plus a new test
asserting on the raw address and its percent-encoded form, turns the
suite red: the valueless-query-parameter defect (redact.py) currently
lets ?victim@example.org through as a kept parameter NAME. Left
failing on purpose; the next commit fixes redact.py and turns it
green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
|
|
Same arrangement as qtmaildir: one source of truth, and a thin pointer
beside it so every agent tool reads the same file.
It leads with the three properties that are not negotiable, because each
has a concrete victim and each is a thing a later change could quietly
break. Recipient identifiers must never reach a report, and the guarantee
is structural rather than a step someone remembers. Nothing is fetched or
resolved, which the suite verifies by running with sockets disabled. The
trust boundary is configured rather than guessed, and its test is
mutation-checked because walking one hop too far reports an innocent party
the attacker named.
It also records the traps that were found the hard way rather than
reasoned about: ipaddress.ip_network(42) returning 0.0.0.42/32 instead of
raising, a string trusted_relays iterating characters, goog.json looking
like the right SPF source while listing all of Google, and the provider
table having been wrong in every entry when it was first written from
memory.
The hand-testing rule is stated with its evidence: the prompts are the
user's to test, and one pass over them found five defects, four being the
same mistake of validating an answer somewhere other than where it was
given.
Every factual claim in it was checked against the code before committing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
|
|
Marks what is built rather than describing the whole pipeline as though it
existed, documents the three ways to answer the trust-boundary question,
and shows what a case directory holds and what an origin and confidence
mean on an indicator.
Every command in it was run before committing, including the two
non-interactive forms.
The tests section names the two checks that are not ordinary unit tests,
because they are the ones a reader would otherwise not know to keep: the
Received-chain mutation check, and the parser suite running with sockets
disabled.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
|
|
Five defects from the user's hand test of the interactive setup, four of
them the same mistake: an answer was validated somewhere other than where
it was given, so a typo cost the whole run.
A bad CIDR was only rejected after the NEXT question had been asked and
answered, and both answers were then discarded. It is validated at its own
prompt now and re-asks, so a mistake costs one retry.
A provider name typed at the first prompt was treated as a CIDR, so
"Gmail" produced "does not appear to be an IPv4 or IPv6 network", which
names nothing the user can act on. The prompt takes either answer now and
tries the provider table first, because the user does not know, and should
not have to know, which of the two questions they are answering.
Picking a hop number out of range raised a bare IndexError, and a
non-numeric pick leaked int()'s own message about base 10. Both re-ask
naming the valid range.
The first question said what to type without saying what it was for. It
now states that the boundary decides which address gets reported, which is
the fact that makes a wrong answer matter.
Ctrl+C and EOF are caught in main() and report that nothing was written.
Setup writes nothing until every answer is in hand, so abandoning it
midway leaves no half-written config, and that was already true; it just
said so with a traceback.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
|
|
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
|
|
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
|
|
An empty cases string resolved to Path("") = cwd, scattering evidence
wherever the command happened to run. A string trusted_relays (easy to
hand-write without brackets) iterated as characters, failing on '1'
with an error naming nothing findable in the file.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
|
|
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
|
|
The IOC assembly needed each URL in two forms: redacted for the report,
original to recognise a suspect token, since by the time a query value
reads REDACTED there is nothing left to look at. That arrived as a second
scanner rebuilding a {redacted: original} mapping by repeating urls()'s
own walk, which is two functions that have to stay in step by hand.
One private _scan_urls() returns both forms instead. urls() keeps its
signature and is now a sort over its keys, and the two forms cannot drift
because nothing derives one from the other twice.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
|
|
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
|
|
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
|
|
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
|
|
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
|
|
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
|
|
The query string was not the only place a recipient identifier can hide.
A fragment (#e=victim@...) is published as-is since we report the URL's
literal text, not what a browser would send. Userinfo (user:pass@host)
leaks a credential as well as an identifier, so it is stripped outright
rather than redacted in place. A path segment can also smuggle an
encoded query (%3Fe=victim@...); suspect_path_segments now flags a
segment that decodes to something containing '=' or '@', still leaving
the decision to redact or not to human review.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
|
|
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
|
|
Four hand-written messages using example.org, example.invalid and the
RFC 5737 documentation IP ranges. No real phishing sample goes in this
repository: it would carry the recipient identifiers this tool exists to
keep out of reports, and a repository is potentially public.
forged-chain.eml is the one that matters. The attacker prepends two
Received headers naming an innocent third party, so a parser that walks
past the trust boundary reports 198.51.100.7 rather than 203.0.113.99.
Weekdays verified with date(1) rather than written from memory, since an
RFC2822 parser validates the day against the date and a wrong one is
indistinguishable from a malformed header.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
|
|
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
|
|
Its SPF is a tree of includes rather than a flat list: spf.privateemail.com
carries no addresses at all, only includes, and one branch nests a further
level. Two of the branches live on jellyfish.systems. The entry here is the
flattened union of the five leaf records, deduplicated, 18 networks.
The re-verification command lists the leaves rather than the top-level
name, and a note says why: querying spf.privateemail.com and finding no
ip4 entries looks like a stale record and is not one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
|
|
Every range in the first draft was wrong. They had been written from
memory, and the correct source is each provider's own SPF record, which
is the list of addresses it declares it sends from.
Gmail is the clearest case: the table claimed eleven IPv4 ranges and
_spf.google.com publishes two. Fastmail, Proton and the rest were wrong
in the same way. Outlook and Zoho are added since they were queried
anyway, and the transcription commands are recorded in a comment so the
next check is a copy-paste rather than a search.
Google's goog.json is deliberately NOT the source, though it looks like
one: it lists all Google infrastructure, over a hundred ranges, and using
it would trust every Google-hosted service as part of the user's own mail
path. _spf.google.com is the mail-sending answer.
The table is IPv4 only. An IPv6 hop from one of these providers does not
match and the user is asked instead, which is the safe direction: a hop
wrongly trusted means the real sender is never reported.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
|
|
Thirteen tasks, TDD throughout, stdlib only. parse is pure and offline:
the trusted-relay boundary arrives as an argument rather than a config
read, so the whole extractor is testable against fixtures with no setup.
The plan carries three checks that are not ordinary unit tests. The
Received-chain task has a mutation step, because walking one hop too far
reports an innocent third party named in a header the attacker wrote, and
a test that cannot fail would not protect against it. The URL task runs
the suite with sockets refused, so the never-fetch rule is verified
rather than read. And every fixture is asserted to leave no recipient
address anywhere in the manifest.
init exists because parse refuses to guess the trust boundary. It asks
for CIDRs, offers a static table of known provider ranges, or reads the
chain of a known-good sample and lets the user pick their own hops. A
pure builder with the prompts and the flags as two front ends over it, so
--non-interactive covers agent-driven setup and the config writing is
tested without a terminal. Re-running shows what is already configured
and asks; either route backs the old file up first and preserves sections
this run does not set, so a later init cannot silently drop a MISP key.
The prompts themselves are hand-tested rather than driven from stdin: a
test there would assert the wording it was written against and break on a
rewording that improved it. Task 12 is the checklist, weighted towards
wrong answers.
Redirect chains were missing from the first draft of the plan and are now
specified: a parameter whose value is itself a URL is recovered as an
indicator while every other value stays redacted, which resolves the
conflict between reporting the destination and never publishing a
tracking token.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
|
|
A rate limit tells us when, not merely that: a 429 carries Retry-After or
the vendor's reset headers. Recording the deadline and asking the user to
run submit again throws that away and relies on them remembering.
So `abusectl retry` scans every case for destinations whose retry_after
has passed and sends only those, as one unattended cron line beside
mailsync.sh. No inline retry: submit never sleeps waiting for a window,
because a daily quota resets in hours and a process killed while
sleeping is back to the user remembering. One mechanism, not two.
Unattended retry makes three properties load-bearing, since a retry that
re-sends is a duplicate abuse report and cannot be withdrawn. Status is
written before the attempt, so a crash mid-send leaves in-flight, which
is honest, rather than looking like it never happened; retry never
touches in-flight. Attempts are capped, so a dead abuse mailbox stops
being retried. A soft failure with no server deadline gets exponential
backoff.
deferred and failed are separate statuses: deferred means the tool will
handle it, failed means the user must. Collapsing them either strands a
rate-limited report forever or retries a dead mailbox indefinitely.
The qtmaildir dialog accordingly grows no retry button. A deferred
destination belongs to cron, and a button beside it would race the
scheduled run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
|
|
An abuse reporting sidecar for phishing mail: parse a flagged message,
extract its indicators, resolve abuse contacts, and file the result to a
MISP instance and to public abuse channels.
This is the umbrella spec, agreed in one design session. Each part gets
its own spec before it is built; this settles what the parts share and
what would be expensive to change later: the case directory and its
manifest format, the redaction rule, the ordering between MISP and the
vendors, and how partial failure is recorded.
It exists as a separate tool because qtmaildir does no network protocol
work by design, and this needs RDAP, three vendor APIs and mail to abuse
desks. qtmaildir invokes it by name the way it invokes mailsync.sh, and
hosts the review dialog; the two are coupled only by the manifest format
and a command name in config.
Two properties are recorded as safety properties rather than
preferences. Recipient identifiers are never captured, at extraction
rather than at submission, so the tool cannot disclose an identifier it
was never given; tracking tokens inside URLs are covered, since a
parameter value is frequently the recipient's address. And nothing
remote is fetched while parsing, because following a link confirms the
address is live and fires the tracker.
parse is the first part to build: stdlib only, no network, no config,
and its output is the format every other part reads.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
|
