path: root/docs/BACKLOG.md

# Backlog

Open items, newest last. One numbering sequence; a closed item keeps its
number and gains a status rather than being renumbered.

| # | Item | Size | Status |
|---|------|------|--------|
| 1 | Skip boilerplate namespace URLs | XS | open |
| 2 | An IDN indicator resolves to no contact | S | open |
| 3 | Expose kept cases so qtmaildir can tag spam | ? | open, unsized |
| 4 | `Report-Type: phishing` is unverified against x-arf | XS | closed |
| 5 | `report.build()` raises KeyError on an identity with no email | XS | closed |
| 6 | VirusTotal as a destination, or as enrichment | S | open, deferred |

## 1. Skip boilerplate namespace URLs

**Observed.** A sweep of 82 real spam messages reported 8 URL IOCs pointing
at `www.w3.org`, from four distinct values:

```
http://www.w3.org/1999/xhtml
http://www.w3.org/TR/html4/loose.dtd
http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd
https://www.w3.org/1999/xhtml
```

**Cause.** `_scan_urls()` scans the body text for anything URL-shaped, and an
XHTML doctype or namespace declaration is URL-shaped. The spammer did not put
those there; the HTML boilerplate did.

**Approach.** A hostname skip-list, checked after the URL is extracted. Four
hosts cover everything the corpus produced: `www.w3.org`, `schemas.microsoft.com`,
`purl.org`, `ns.adobe.com`. Keep it a list of HOSTS, not a regex over the URL:
a hostile URL can put any string in a path or query, so matching on anything
but the host lets an attacker suppress their own indicator.

**Constraints.** Not a leak and not a correctness defect: it is noise in a
report a human reads, and the cost is filing `w3.org` as phishing
infrastructure with an abuse desk. Weigh against the argument for leaving it
alone, which is real: a parser that reports exactly what the message contained
is easier to defend than one that decides what to omit, and every entry on a
skip-list is a thing an attacker knows will not be reported. If it is built,
the skipped URLs should still be visible somewhere during review rather than
silently dropped, on the same reasoning that makes `suspect_path_segments`
flag rather than redact.

## 2. An IDN indicator resolves to no contact

**Observed.** `contacts.is_queryable()` refuses any non-ASCII value, so a
phishing domain in an internationalised name is recorded as `unusable` with the
reason "not ASCII, and we do not guess at an IDN encoding", and no registry is
asked. The `ponytail:` comment on that function names this as the ceiling it
accepted.

**Cause.** Deliberate, not a defect. Converting a name to punycode means
choosing an encoding for attacker-supplied text, and a wrong choice sends a
different name than the one in the message to a registry, which is a disclosure
made about the wrong party. Refusing keeps the value in front of the user
instead.

**Approach.** `value.encode("idna")` is the obvious move and is not enough on
its own: it normalises, so the name queried may differ from the name written in
the message, and the manifest must record BOTH, the way `query_domain` already
records `queried` separately when the label walk shortens a host. The refusal
reason already distinguishes non-ASCII from illegal, so the user-facing half
exists.

**Constraints.** Not a leak: the failure direction is asking nobody, which is
safe. Weigh against a real argument for leaving it: a homograph name is exactly
where the attacker wants the tool to normalise on their behalf, and a
consultant chasing one indicator by hand is a smaller cost than a query made
about a name the user never saw. Wait for a real IDN indicator in a sweep before
building it.

## 3. Expose kept cases so qtmaildir can tag spam

**Source.** The author's idea note, not a defect found in the code. Unlike
items 1 and 2 the cause here has NOT been verified against the code, because
there is nothing built yet to verify: this is a feature request, and it is
recorded unsized on purpose.

**Observed.** Case directories are permanent by design, so over time they
become a local corpus of messages the user has already judged to be phishing.
Nothing reads them back. The idea is that qtmaildir could ask this tool
whether an incoming message resembles one, and tag it as spam when it does.

**Approach.** Undecided, and the shape matters more than the code. The
umbrella design already fixes the coupling between the two repositories: the
manifest format and a command name in qtmaildir's config, with no submodule.
A read-only subcommand answering a question about one message fits that
contract; a daemon, a socket or a shared database does not, and the umbrella
design rules out a database of this tool's own.

**Constraints, and the real tension.** Deciding a message is spam by
resemblance is a classifier, and this tool has so far been deliberately
mechanical: it reports what a message declared, and refuses rather than
guesses when the trust boundary is unset. A resemblance score is the first
thing here that would be an opinion rather than an observation, and a wrong
one either hides real mail or teaches the user to distrust the tag.

There is also a quieter question about what a match is allowed to be based on.
The obvious signals are the ones already in a manifest, a sending IP, a
domain, a URL shape, an attachment hash. Those are safe. Matching on the
message body would mean holding attacker-supplied text against new mail, and
`source.eml` is unredacted, so anything built here must not become a route by
which a stored recipient identifier reaches a comparison that is later
reported or logged. Property 1 governs what may be published, and a tag is not
a report, but the path from one to the other is short.

**Before building.** Ask the author what "fits certain requisites" means to
him concretely, since that phrase is doing all the work in the note, and
whether he wants a judgement or only the facts, for instance a subcommand that
answers "this IP appears in three kept cases" and leaves the tagging decision
to qtmaildir. The second is much more in keeping with the rest of the tool.

## 4. `Report-Type: phishing` is unverified against x-arf

**Closed** against the IANA MARF registry
(`https://www.iana.org/assignments/marf-parameters/marf-parameters.xml`),
checked 2026-09-10, plus RFC 5965 itself. The primary source the item asked
for exists; it is IANA's, not x-arf's.

Two things were wrong, and neither was a leak.

`Report-Type` is NOT a registered field. The registry lists 25 field names
and it is absent. It came from the worked example in the report spec rather
than from a decision. It is now REMOVED: `Feedback-Type` already carries what
it was saying. Keeping it would have been harmless, since RFC 5965 section 6
makes ignoring an unknown field a MUST for the receiver, but that same section
requires an extension field be registered, and a desk should be able to look
up every field in a document this tool sends.

`Feedback-Type` was `abuse` and is now `fraud`. RFC 5965 registers `fraud` as
"indicates some kind of fraud or phishing activity" and `abuse` as
"unsolicited email or some other kind of email abuse". This tool reports
phishing, so `fraud` is the registered value that says what the report is,
and some desks route it separately from bulk spam.

A third, cosmetic, finding: the module emitted `Reported-Uri` while the RFC's
ABNF and its IANA registration both spell it `Reported-URI`. The RFC's own
worked example writes `Reported-Uri`, which is where the module's spelling
came from. Field names are case-insensitive per RFC 5322 so nothing was
broken, and it now matches the normative spelling.

`tests/test_report.py::FeedbackPart::test_no_unregistered_field_is_emitted`
holds the registry's 25 names and asserts every emitted field is one of them.
The single exception is `Source`, which is x-arf's and is the reason the
envelope carries x-arf fields at all; it is named in the test so a NEW
unregistered field cannot arrive unnoticed.

## 5. `report.build()` raises KeyError on an identity with no email

**Closed** by the `report` subcommand (Task 11). `cli._cmd_report()` refuses
before building, with the not-configured exit code and a sentence naming the
file to edit, rather than letting `build()` raise. The rule it applies is
EMAIL REQUIRED, NAME AND ORG OPTIONAL: the address becomes the `From` and a
report without one cannot be answered, while the other two are individually
skippable at `init` and `text_part()` already renders whatever subset is
present. `tests/test_cli.py` covers both halves, the refusal and the
email-only identity that must still succeed.

**Observed.** `report.build(manifest, destination, identity)` reads
`identity.get("name", "")` defensively but `identity["email"]` directly, so an
identity carrying a name and no address raises `KeyError: 'email'` rather than
saying what is missing. Reproduced through the public API against the real
config reader: `config.load()` on a file whose `[reporter]` sets `name` and
omits `email` returns `{"name": "A Reporter"}`, and that dict raises.

**Cause.** Every key in `[reporter]` is independently optional, by the same
skipped-answer-is-absent rule the rest of the config follows, but the report
builder treats one of them as required without checking. Task 9 fixed the
config half of this seam; the report half was not reachable from a config file
until `init` grew the three prompts, and now it is: skipping the email
question while answering the name produces exactly this shape.

**Approach.** Refuse before building, not after: raise a named error saying no
reply address is configured. Filling in an empty addr-spec instead would be
worse, since a `From:` with no address produces a report that is sent and
cannot be replied to, which defeats the reason the identity is disclosed at
all. `init` now warns at the prompt when the address is skipped, so the
remaining gap is a hand-edited config and the error message it deserves.

**Constraints.** Belongs with the `report` subcommand rather than the builder
alone, since where the check lives decides whether the user sees an exit code
and a sentence or a traceback. Not a leak: the failure is loud and nothing is
sent.

## 6. VirusTotal as a destination, or as enrichment

**Deferred**, not rejected. Recorded with the findings so it is reconsidered
from checked documentation rather than from memory.

**Observed.** The report spec's addendum settles three destinations, MISP,
AbuseIPDB and URLhaus. VirusTotal was examined in the same pass and left out.

**What the documentation says**, read 2026-09-10 from `docs.virustotal.com`
and the endpoint index at `virustotal.readme.io/llms.txt`:

- The only submission endpoints are `POST /urls` and file upload. **There is
  no endpoint that submits a domain or an IP address as a new indicator.**
  `POST /ip/{ip}/analyse` and `POST /domains/{domain}/analyse` re-analyse data
  VirusTotal already holds, which is not reporting.
- Files are submitted as FILES. This tool holds an attachment's sha256 and
  never the attachment, so a hash IOC earns no VirusTotal submission either.
- The auth header is `x-apikey`.

So its submit side accepts URLs and nothing else, which is exactly URLhaus's
accepted type. There is no case where VirusTotal earns a row and URLhaus does
not.

**Why it was left out.** The overlap above, plus a difference in kind:
`POST /urls` means "scan this", not "this is malicious". It carries no verdict,
no category and no comment, so the argument the x-arf body makes, what happened
and what is wanted, has nowhere to land. Recording a judgement would mean a
second call to `POST /urls/{id}/votes` with different semantics. URLhaus is
purpose-built for reporting a malicious URL and takes a threat classification
and tags.

**The interesting half is the READ side, and it collides with a property.**
`GET /domains/{d}` and `GET /ip/{ip}` answer questions review actually has:
how old is this domain, has anyone else flagged this host, is this dedicated
attacker infrastructure or a compromised legitimate one. That would change
what a user writes in a report and whether they send one.

Two obstacles, and the second is the real one:

- The second property says nothing is ever fetched or resolved. A VirusTotal
  lookup asks a third party ABOUT an indicator rather than fetching it, so no
  tracker fires and nothing confirms the address is live. That is the same
  shape as an RDAP query, which the tool already makes. The property survives.
- The fourth property is the obstacle. A query discloses which host is being
  investigated, and `contacts.is_queryable()` is the single admission point
  that exists because that leak happened three times. **A URL lookup cannot
  pass it**: VirusTotal's URL endpoint takes a URL, and the fourth property is
  precisely that a query carries a bare host or IP and never a URL. Redacting
  the URL first does not rescue it, since a redacted URL hashes to a different
  `/urls/{id}` than the real one and the lookup returns nothing. Domain and IP
  lookups are bare values and would be fine.

There is also a preference cost: VirusTotal is Google-owned, and every lookup
tells Google which infrastructure a named consultant is investigating. RDAP
carries the same disclosure but is unavoidable if the tool wants an abuse
contact; this would be optional convenience.

**Before building.** Use the tool on real cases first and find out whether
review actually feels under-informed. Enrichment built against a guess about
what review will want is the first rung of the ladder failing. If it is built,
it is a `contacts`-time enrichment over domains and IPs through the existing
admission point, writing what it learns into the manifest, and it needs its
own spec: it changes a module that is already swept and settled.