diff options
| -rw-r--r-- | AGENTS.md | 38 | ||||
| -rw-r--r-- | docs/BACKLOG.md | 40 |
2 files changed, 78 insertions, 0 deletions
@@ -230,6 +230,41 @@ weekday with `date -d <yyyy-mm-dd> +%A` rather than writing it from memory: an RFC2822 parser validates the day against the date and a wrong one reads as a malformed header. +**Sweep the user's real spam as a HAND TEST, and never as a fixture.** A +synthetic fixture only ever contains the shapes someone thought of, and the +shapes nobody thought of are the ones that leak. Real mail is the only source +of those, so the leak property is checked against `tag:spam` in the user's own +notmuch index. It is worth doing: the first message swept found a header the +parser did not read at all, and the corpus exercised a spoofed `Reply-To` +display name that no fixture had. + +Ask before reading the user's mail. Then, from a script in the scratchpad and +never in the repo: + +```python +raw = subprocess.run(["notmuch", "show", "--format=raw", mid], + capture_output=True, check=True).stdout +iocs = parse.iocs(raw, trusted=[...]) +blob = repr(iocs) +for addr in set(re.findall(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+", + raw.decode("utf-8", "replace"))): + assert addr not in blob, (mid, addr) +``` + +The assertion is the whole point and it must be that broad: compare EVERY +address in the raw source, headers and body, against the entire IOC output. +Checking only the recipient misses an address the parser invented from a +display name, which is exactly how the `_domain_of` defect reached a report. +Also count crashes and empty results; a message that parses to nothing is a +finding too. + +What may leave that script: counts, header names, origin tallies, and a +domain with its local part removed. What may not: an address, a subject, a +Message-ID, a URL from a real message, or a real domain written into a test. +When a sweep finds a defect, reproduce it as a synthetic fixture under the +rule above and commit THAT. The real message stays in the scratchpad, which +is per-session and outside the repository. + ## Working on this repo Work directly on `master`, no PR flow. Commits are GPG-signed (`git commit -S`); @@ -249,3 +284,6 @@ umbrella design settles only what they share. ordering between MISP and the vendors. - `docs/plans/2026-09-08-parse.md` — the plan `init` and `parse` were built from. Historical once built, but it records why each test exists. +- `docs/BACKLOG.md` — open items, with the cause verified in the code rather + than assumed. Read it before starting work; add to it rather than fixing + something unasked. diff --git a/docs/BACKLOG.md b/docs/BACKLOG.md new file mode 100644 index 0000000..25793e1 --- /dev/null +++ b/docs/BACKLOG.md @@ -0,0 +1,40 @@ +# Backlog + +Open items, newest last. One numbering sequence; a closed item keeps its +number and gains a status rather than being renumbered. + +| # | Item | Size | Status | +|---|------|------|--------| +| 1 | Skip boilerplate namespace URLs | XS | open | + +## 1. Skip boilerplate namespace URLs + +**Observed.** A sweep of 82 real spam messages reported 8 URL IOCs pointing +at `www.w3.org`, from four distinct values: + +``` +http://www.w3.org/1999/xhtml +http://www.w3.org/TR/html4/loose.dtd +http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd +https://www.w3.org/1999/xhtml +``` + +**Cause.** `_scan_urls()` scans the body text for anything URL-shaped, and an +XHTML doctype or namespace declaration is URL-shaped. The spammer did not put +those there; the HTML boilerplate did. + +**Approach.** A hostname skip-list, checked after the URL is extracted. Four +hosts cover everything the corpus produced: `www.w3.org`, `schemas.microsoft.com`, +`purl.org`, `ns.adobe.com`. Keep it a list of HOSTS, not a regex over the URL: +a hostile URL can put any string in a path or query, so matching on anything +but the host lets an attacker suppress their own indicator. + +**Constraints.** Not a leak and not a correctness defect: it is noise in a +report a human reads, and the cost is filing `w3.org` as phishing +infrastructure with an abuse desk. Weigh against the argument for leaving it +alone, which is real: a parser that reports exactly what the message contained +is easier to defend than one that decides what to omit, and every entry on a +skip-list is a thing an attacker knows will not be reported. If it is built, +the skipped URLs should still be visible somewhere during review rather than +silently dropped, on the same reasoning that makes `suspect_path_segments` +flag rather than redact. |