aboutsummaryrefslogtreecommitdiffstats
path: root/tests
AgeCommit message (Collapse)AuthorFilesLines
36 hoursfix: percent-encode the component interpolated into an RDAP queryDanilo M.1-0/+89
contacts.is_queryable() is the admission point and remains the real gate, but this property has leaked three times already, each time the same shape: a validator applied to one branch and forgotten on its sibling. First the url branch was guarded and domain leaked, then domain was guarded and the IP branch leaked through an IPv6 scope id. query_ip and query_domain interpolated their component straight into the path, so a caller reaching them without passing through worklist() could put a separator, a query or a fragment on the wire. safe="" rather than urllib's default safe="/" is the point: the default leaves the path separator intact, which is exactly the traversal shape that leaked before. query_domain still returns the UNQUOTED candidate, because that is what the manifest records and the review dialog shows. The tests call both functions directly, bypassing contacts, since a future branch that skips the admission point is the failure this layer exists to survive. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
36 hoursfix: refuse an ipv6 scope id, and say why a value was refusedDanilo M.1-2/+63
Four holes of one class, all in the path that admits a query. ipaddress.ip_address accepts a scope id since Python 3.9, so "fe80::1%victim@example.org" passed _is_ip, is_queryable returned on that branch without inspecting the string further, and rdap.query_ip interpolated the whole thing into the query URL unquoted. A recipient identifier reached a registry, which is exactly the disclosure the fourth property exists to prevent and exactly what the domain-branch fix closed. Reject a scope id at the admission point. fe80::/10 is link-local and never a legitimate RDAP target anyway. The ip branch of worklist() dropped an unparseable value silently rather than funnelling it through add() like the other two branches, so a mangled indicator vanished from the manifest instead of showing up as unusable. It now goes through add(), and is_queryable grew the check that makes that classification correct: an all-digit last label is never a TLD, so "999.999.999.999" is a malformed IP rather than a host to ask a registry about. Two behaviours had no test and both mutations survived. _host_of's root-dot strip is now covered through the url branch, since failing to fold discloses one host to a registry twice; the .lower() half was dead work because urlsplit already lowercases, and is dropped with a comment saying so. The two-label requirement is asserted on is_queryable directly, because a single-label name matches no tld in the bootstrap and a resolve test could never fail whatever the validator decides. The refusal message named no reason, leaving a reviewer unable to tell whether the parser mangled a legitimate host or the attacker planted something. Two cases now: a non-ASCII name is plausibly a real IDN indicator to chase by hand, an illegal character is hostile. Also let KeyboardInterrupt and SystemExit through resolve()'s handler, so stopping a long run is not recorded as a failed query on whichever indicator was in flight. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
36 hoursfix: validate a query host at the one point a query is admittedDanilo M.1-0/+98
THE FOURTH PROPERTY was breached through the domain branch of worklist(). The url branch is cleaned by _host_of, which uses urlsplit().hostname, and the domain branch did only value.strip(".").lower(). rdap.query_domain then interpolates that value into the fetch URL with no quoting. parse._domain_of takes everything after the @ of a From, Sender or Reply-To addr-spec, and email.utils.parseaddr permits /, ?, # and % there, so the whole shape is attacker-controlled through a header they own. A From of `Bank <phish@victim%40example.org.invalid>` sent the recipient's own address to a registry, which is precisely the identity disclosure the property exists to prevent, reaching a third party. `a/../../x.invalid` escaped the /domain/ endpoint altogether, and query, fragment and space values all reached the wire. Fixed at the SINGLE admission point rather than in the offending branch, because per-branch validation is what failed here: one branch was cleaned, the next was written without it. is_queryable() now guards worklist()'s add(), so domain, url and any future branch pass through it. A rejected value is not dropped. It keeps an entry with an empty abuse list and an error saying it was never queried, following the rule suspect_path_segments already sets: flagged and visible to the user during review, because silent was the bug. A non-ASCII host is refused rather than encoded to punycode. Guessing the encoding of an attacker-supplied name is a query that cannot be justified, and the ceiling is noted in a ponytail comment. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
36 hourstest: make two rdap tests able to fail, and cover the hostile bootstrapDanilo M.1-2/+106
test_the_walk_is_capped could not fail. Its host had nine labels, so the bare-TLD guard alone bounded the walk to eight attempts and the cap was never reached, and the assertion read rdap._MAX_LABEL_WALK at assert time, so the expected value moved with any mutation. Raising the cap to a billion left the suite green. It now uses a twelve-label host and a hard-coded expected count of five, and raising the cap fails it with 11 != 5. test_a_newline_in_an_address_is_rejected asserted a guarantee that a different function happened to provide. email.utils.parseaddr rejects the CRLF fixture by itself, returning an empty address, so deleting both control-character checks from _valid_address left the suite green while the real defence was gone. parseaddr passes \x0b and \x0c straight through, and those reach a mail header raw, so a case for each is added alongside the CRLF one, which is still worth asserting. The new HostileBootstrap cases cover the bootstrap defects fixed in the previous commit: a planted integer CIDR, a malformed services value in seven shapes, a plaintext base URL, and the two paths where a bad entry must not hide a good one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
36 hoursfeat: resolve the worklist to abuse contactsDanilo M.1-0/+126
Failure is per query and never stops the run: a timeout on one indicator must not cost the contacts that did resolve, and a missing contact is a normal outcome rather than an error. Bootstraps are passed in rather than fetched here, so this stays testable with no network and the caller owns the cache policy. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
36 hoursfeat: build the contacts worklist, host onlyDanilo M.1-0/+113
Adds the fourth non-negotiable property: a query carries a bare host or IP and never a URL. An RDAP query discloses what the user is looking at, and a URL path can carry recipient identity that parse deliberately flags rather than redacts, because a path segment may be the thing being reported. That decision is safe only while the URL stays local. Property 1 governs what is published and a query appears in no report, so property 1 does not cover this and this property does. Hosts fold, so twenty URLs on one host make one query while the item keeps every indicator id behind it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
36 hoursfeat: query RDAP, walking up the labels for a registrable domainDanilo M.1-0/+80
RDAP wants the registrable domain and a deep host is not one. Rather than bundling a Public Suffix List, which is a transcribed table that goes stale weekly and is the failure init.PROVIDERS already documents, this asks the registry, which is the authority on what is registrable. The walk is capped and never queries a bare TLD. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
36 hoursfeat: read abuse addresses from a jCard, strictlyDanilo M.1-0/+85
Only an entity whose roles contain abuse counts. No fallback to a technical or registrant contact, who is a named human that never volunteered for abuse mail, and no fallback to abuse@<domain> by convention: for a phishing domain that mailbox belongs to the attacker, so constructing it would confirm both the catch and that the reporter's address is live. Addresses are validated where they enter rather than where they are used, because the value becomes a mail recipient later and a control character in it is header injection into mail this tool sends. Entity recursion is depth capped so remote JSON cannot hang the tool. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
36 hoursfeat: select an RDAP server by longest prefix and by TLDDanilo M.1-0/+51
Longest prefix rather than first match: a block delegated to a new operator appears as a more specific range inside its parent, and the wider range would name the operator that gave it away. A TLD that publishes no RDAP service selects nothing, which is a normal outcome for many TLDs rather than a defect. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
36 hoursfeat: cache the IANA bootstrap, and prefer a stale copy to noneDanilo M.1-0/+73
Seven day TTL under XDG_CACHE_HOME, deliberately not in a case directory: this is a copy of a public map, not evidence. A failed refetch falls back to the stale copy. Staleness is safe in this direction, since a server that has moved simply misses and reads as no contact, while losing IANA entirely would stop the user filing a report. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
36 hoursfeat: add the RDAP transport, with a redirect cap and no downgradeDanilo M.1-0/+56
The only socket in this tool. A redirect is remote data directing our next request, so hops are capped and an https to http downgrade is refused: a downgraded query travels in clear text and discloses which netblock is under investigation to anyone on the path. The timeout is mandatory rather than defaulted, because urllib with no timeout blocks forever and a hung registry would hang a review. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wrfqr2xqQfhtXCscU7zrdz
2 daysfeat: report List-Unsubscribe urls and a differing SenderDanilo M.2-0/+19
A sweep of the user's real spam found List-Unsubscribe naming a domain that appeared nowhere else in the message. It is attacker infrastructure and was going unreported. Every url from that header goes through redact.url() like a body url: an unsubscribe link has to say who is unsubscribing, which makes it one of the likeliest carriers of a recipient token. mailto: entries are skipped rather than redacted, since the address is the whole value and nothing useful survives removing it. Sender is collected on the same terms as Reply-To, included only when it differs from From. One repeating From is noise; one naming a separate relay is the infrastructure behind the run. Also drops the unused urlencode import left in redact.py when _redact_kv_string stopped using urllib to rebuild the query string. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019NHaqA1Rz5ybed7wFUeQbK
2 daysfeat: flag an address spoofed into a display nameDanilo M.1-0/+16
A display name naming a recognisable address is a deliberate act and a real signal, so it is reported rather than merely ignored once _domain_of() stopped mistaking it for the sender. The IOC carries no value. One of the identities an attacker impersonates is the recipient themselves, so publishing the impersonated domain would leak the recipient's own domain in exactly the case worth flagging. What travels is only that it happened; the reviewer has the message and can see who was impersonated, and a third party does not need to. display_name_addresses() still returns the full address for local review. Only the published IOC list is stripped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019NHaqA1Rz5ybed7wFUeQbK
2 daysfix: take the sender domain from the address, not the display nameDanilo M.2-1/+8
_ADDR_DOMAIN.search() returned the first @domain anywhere in the raw header text. A display name sits before the angle brackets and is attacker-controlled, so it won. Two ways that reached a published report. The sender was misattributed: "Billing at billing@innocent.example" <phish@sender.example.invalid> filed the report against a third party who sent nothing. And it defeated the structural guarantee in sender_domains(): the module reads no recipient header, but an attacker who writes the victim's own address into the display name hands it one anyway, and it came back out as a sender domain. parseaddr() parses the header grammar rather than scanning it, so a quoted display name cannot supply the address. leaky.eml's display name now carries the recipient address. The existing test_no_ioc_holds_a_recipient_address assertion catches this class; it was green before only because the fixture used a harmless domain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019NHaqA1Rz5ybed7wFUeQbK
2 daysfix: prefer the observed address over an attacker's HELO literalDanilo M.1-0/+22
_extract_ip returned the FIRST bracketed IP in a Received header value. Postfix (and others) write the client's own HELO/EHLO argument first and the address it actually observed on the connection second: Received: from [198.51.100.7] (unknown [203.0.113.99]) by mx... The first bracket is entirely attacker-chosen; a client can HELO with any literal it likes. sending_ip() returned 198.51.100.7, reporting whoever the attacker named rather than 203.0.113.99, the address the accepting server itself wrote. This needs no forged extra hop, only a client that HELOs with an address literal, and the module's own docstring already stated the intended answer ("the bracketed literal after the connecting hostname") without the code implementing it. _extract_ip now collects every bracketed, ipaddress-valid literal with its position and, when there is more than one, prefers the last one appearing before " by " (the accepting server's own clause, and the one closest to it). A header with a single bracketed IP or no " by " token keeps the previous single-candidate behaviour, so simple.eml (203.0.113.42) and forged-chain.eml (203.0.113.99, item 3's own mutation-checked test) are unaffected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
2 daysfix: re-bracket an IPv6 host when stripping URL userinfoDanilo M.1-0/+12
parts.hostname returns an IPv6 literal WITHOUT its brackets (2001:db8::1, not [2001:db8::1]), and _netloc_without_userinfo reassembled f"{host}:{port}" directly from it: redact.url("http://victim@[2001:db8::1]:8080/p?x=1") -> "http://2001:db8::1:8080/p?x=REDACTED" That string is not parseable back into a host and a port, and the digits after the second-to-last colon are not even part of the address any more. Reporting it means the abuse desk cannot identify the host at all, or misreads it, which is the same class of harm as reporting a wrong IP outright. _netloc_without_userinfo re-adds brackets whenever the hostname contains ":", so an IPv6 host now survives userinfo stripping exactly as an IPv4 or named host already did. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
2 daysfix: redact a valueless query token whole, not as a kept nameDanilo M.1-0/+14
parse_qsl reads "?victim@example.org" as the pair ("victim@example.org", ""), and the code kept parameter NAMES because they fingerprint the phishing kit. A name that is itself an address is not a name, so keeping it published the recipient's identifier verbatim (percent-decoded, no less: %40 fools no consumer). _redact_kv_string now splits the query/fragment string by hand on "&" and ";" and inspects each token's own name: one containing "@" is a value that landed in name position and is redacted WHOLE ("?REDACTED" rather than "?victim%40example.org=REDACTED"); an ordinary name still keeps its shape ("?flag" stays "?flag=REDACTED", "?t=1&t=2" stays "?t=REDACTED&t=REDACTED"). This also exposed a second leak reachable through the same fixture: parse._suspect_segments() ran redact.suspect_path_segments()'s decode-and-check predicate (meant for a querystring smuggled past percent-encoding into a PATH segment) against a raw query VALUE, so a plaintext "?e=you@example.org" reproduced the address in the manifest's suspect_path_segments flag even though the URL itself was correctly redacted. redact.suspect_path_segments() now exposes the opaque-shape half of its check as _looks_opaque(), and parse.py uses only that half against query values: a query value is always fully redacted regardless, so the flag may hint at its shape but must never reproduce it. Turns tests.test_parse.TestIocAssembly.test_the_address_does_not_survive_any_url_shape green, and closes the gap test_no_ioc_holds_a_recipient_address had been passing over with URL redaction fully disabled. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
2 daystest: wire leaky.eml into the anti-leak suite (red, defect 2)Danilo M.2-1/+29
test_no_ioc_holds_a_recipient_address grepped for the literal "example.org", and every existing fixture hides the recipient address as base64, so URL redaction could be disabled entirely and both this test and test_cli's counterpart stayed green. leaky.eml carries you@example.org in five URL shapes plus a From display-name trap. Adding it to the fixture list, plus a new test asserting on the raw address and its percent-encoded form, turns the suite red: the valueless-query-parameter defect (redact.py) currently lets ?victim@example.org through as a kept parameter NAME. Left failing on purpose; the next commit fixes redact.py and turns it green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
2 daysfeat: command line for init and parseDanilo M.1-0/+167
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
2 daysfeat: first-run config builderDanilo M.1-0/+185
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
2 daysfix: reject non-list relays and empty cases valuesDanilo M.1-0/+29
An empty cases string resolved to Path("") = cwd, scattering evidence wherever the command happened to run. A string trusted_relays (easy to hand-write without brackets) iterated as characters, failing on '1' with an error naming nothing findable in the file. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
2 daysfeat: read and validate the configDanilo M.1-0/+88
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
2 daysfeat: assemble IOCs in the manifest's shapeDanilo M.1-0/+63
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
2 daysfeat: case directory with an atomically written manifestDanilo M.1-0/+102
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
2 daysfeat: extract URLs and attachment hashes, fetching nothingDanilo M.1-0/+61
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
2 daysfeat: extract sender domains and auth verdictsDanilo M.1-0/+32
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
2 daysfeat: walk the Received chain to the trust boundaryDanilo M.1-0/+68
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
2 daysfix: redact fragment and strip userinfo, flag hidden query in pathDanilo M.1-0/+42
The query string was not the only place a recipient identifier can hide. A fragment (#e=victim@...) is published as-is since we report the URL's literal text, not what a browser would send. Userinfo (user:pass@host) leaks a credential as well as an identifier, so it is stripped outright rather than redacted in place. A path segment can also smuggle an encoded query (%3Fe=victim@...); suspect_path_segments now flags a segment that decodes to something containing '=' or '@', still leaving the decision to redact or not to human review. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
2 daysfeat: redact recipient identifiers inside URLsDanilo M.1-0/+106
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
2 daystest: fixtures for the parser, documentation ranges onlyDanilo M.4-0/+81
Four hand-written messages using example.org, example.invalid and the RFC 5737 documentation IP ranges. No real phishing sample goes in this repository: it would carry the recipient identifiers this tool exists to keep out of reports, and a repository is potentially public. forged-chain.eml is the one that matters. The attacker prepends two Received headers naming an innocent third party, so a parser that walks past the trust boundary reports 198.51.100.7 rather than 203.0.113.99. Weekdays verified with date(1) rather than written from memory, since an RFC2822 parser validates the day against the date and a wrong one is indistinguishable from a malformed header. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R
2 daysfeat: package skeletonDanilo M.2-0/+28
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KphFXTc2QajxXsHWyvGJ4R