1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
|
# Backlog
Open items, newest last. One numbering sequence; a closed item keeps its
number and gains a status rather than being renumbered.
| # | Item | Size | Status |
|---|------|------|--------|
| 1 | Skip boilerplate namespace URLs | XS | open |
| 2 | An IDN indicator resolves to no contact | S | open |
## 1. Skip boilerplate namespace URLs
**Observed.** A sweep of 82 real spam messages reported 8 URL IOCs pointing
at `www.w3.org`, from four distinct values:
```
http://www.w3.org/1999/xhtml
http://www.w3.org/TR/html4/loose.dtd
http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd
https://www.w3.org/1999/xhtml
```
**Cause.** `_scan_urls()` scans the body text for anything URL-shaped, and an
XHTML doctype or namespace declaration is URL-shaped. The spammer did not put
those there; the HTML boilerplate did.
**Approach.** A hostname skip-list, checked after the URL is extracted. Four
hosts cover everything the corpus produced: `www.w3.org`, `schemas.microsoft.com`,
`purl.org`, `ns.adobe.com`. Keep it a list of HOSTS, not a regex over the URL:
a hostile URL can put any string in a path or query, so matching on anything
but the host lets an attacker suppress their own indicator.
**Constraints.** Not a leak and not a correctness defect: it is noise in a
report a human reads, and the cost is filing `w3.org` as phishing
infrastructure with an abuse desk. Weigh against the argument for leaving it
alone, which is real: a parser that reports exactly what the message contained
is easier to defend than one that decides what to omit, and every entry on a
skip-list is a thing an attacker knows will not be reported. If it is built,
the skipped URLs should still be visible somewhere during review rather than
silently dropped, on the same reasoning that makes `suspect_path_segments`
flag rather than redact.
## 2. An IDN indicator resolves to no contact
**Observed.** `contacts.is_queryable()` refuses any non-ASCII value, so a
phishing domain in an internationalised name is recorded as `unusable` with the
reason "not ASCII, and we do not guess at an IDN encoding", and no registry is
asked. The `ponytail:` comment on that function names this as the ceiling it
accepted.
**Cause.** Deliberate, not a defect. Converting a name to punycode means
choosing an encoding for attacker-supplied text, and a wrong choice sends a
different name than the one in the message to a registry, which is a disclosure
made about the wrong party. Refusing keeps the value in front of the user
instead.
**Approach.** `value.encode("idna")` is the obvious move and is not enough on
its own: it normalises, so the name queried may differ from the name written in
the message, and the manifest must record BOTH, the way `query_domain` already
records `queried` separately when the label walk shortens a host. The refusal
reason already distinguishes non-ASCII from illegal, so the user-facing half
exists.
**Constraints.** Not a leak: the failure direction is asking nobody, which is
safe. Weigh against a real argument for leaving it: a homograph name is exactly
where the attacker wants the tool to normalise on their behalf, and a
consultant chasing one indicator by hand is a smaller cost than a query made
about a name the user never saw. Wait for a real IDN indicator in a sweep before
building it.
|
