From bc909d9886e0c679fec1589734199de8873f9469 Mon Sep 17 00:00:00 2001 From: "Danilo M." Date: Fri, 31 Jul 2026 08:57:50 +0200 Subject: feat: log permanently failed downloads to failures.json State advances past a download that exhausts MAX_RETRIES so one broken file cannot wedge the backup, which until now left a silent gap in the archive. Failures are appended to failures.json in the archive dir as JSONL, one {id, file, error} per line; appending is a single write, so a crash costs the last record rather than the file. download_with_retry now returns the last exception or None on success, replacing the True/False return, so the caller has an error to log. Nothing reads or prunes the file: it is a record, not a retry queue. Co-Authored-By: Claude Opus 5 --- CLAUDE.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) (limited to 'CLAUDE.md') diff --git a/CLAUDE.md b/CLAUDE.md index 551fc5f..c0af417 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -28,7 +28,7 @@ Three things are load-bearing and easy to break: 1. **`reverse=True` on `iter_messages`** (`tg_backup.py:75`). Processing must be oldest-to-newest. `state['last_id']` is a high-water mark used as `min_id` on resume; iterating newest-first would make it skip unfetched history permanently. 2. **State saved after every message**, not at the end, via temp-file + `os.replace`. Cheap insurance against interruption; do not "optimize" into a batched write without keeping resume correctness, and keep the write atomic. A truncated `state.json` makes the next run unstartable. -3. **Two-level retry handling.** The inner `download_with_retry` retries a single file up to `MAX_RETRIES`: FloodWait sleeps the server-requested interval, other exceptions (network drops, expired file references, disk errors) get exponential backoff so one flaky socket can't kill an overnight run. The outer `while True` in `run_backup` catches FloodWait on history fetching, saves state, sleeps, and restarts the iterator from the saved id. A permanently failed download still advances state on purpose, so one broken file can't wedge the backup forever. +3. **Two-level retry handling.** The inner `download_with_retry` retries a single file up to `MAX_RETRIES`: FloodWait sleeps the server-requested interval, other exceptions (network drops, expired file references, disk errors) get exponential backoff so one flaky socket can't kill an overnight run. The outer `while True` in `run_backup` catches FloodWait on history fetching, saves state, sleeps, and restarts the iterator from the saved id. A permanently failed download still advances state on purpose, so one broken file can't wedge the backup forever; the gap is recorded in `failures.json` (see below) rather than lost. `download_with_retry` returns the last exception, or `None` on success, so the caller has something to log. 4. **Media downloads to `{name}.part`, renamed on success.** The dedup check is `filepath.exists()`, so a truncated file at the final path would be skipped forever as if complete. The `.part` file is unlinked on failure and on interrupt, leaving the message to be retried on the next run. Resume/dedup is also file-based: media is named `{message.id}_{sanitized_name}` (see `safe_filename`, which strips sender-controlled path components) and an existing path is skipped, so state loss degrades to a slow rescan rather than re-downloading everything. @@ -39,7 +39,11 @@ Resume/dedup is also file-based: media is named `{message.id}_{sanitized_name}` ## Layout -`--archive-dir` holds media files and `state.json`, and is created if missing (`parents=True`). It is per-chat: one dir per backed-up chat. +`--archive-dir` holds media files, `state.json`, and `failures.json`, and is created if missing (`parents=True`). It is per-chat: one dir per backed-up chat. + +`state.json` also stores the `target` the dir was last backed up with, so subsequent runs need only `--archive-dir`. An explicit `--target` still wins; it is only required for a dir with no saved target (a first run, or one created before this was added). Nothing guards against pointing an existing dir at a *different* chat: `last_id` from the old chat would be applied as `min_id` to the new one and skip its history. + +`failures.json` is append-only JSONL, one `{"id", "file", "error"}` object per line, written only when a download exhausts `MAX_RETRIES`. JSONL rather than a JSON array so appending is a single write with no read-modify-write: a crash costs the last line, not the file. Read it with `jq -s`. Nothing reads or prunes it, and a message failing on two runs is logged twice. Auth lives in `~/.config/telegram_backup/` (mode 0700): `config.json` with the api_id/api_hash, and `session.session`, a single Telethon session **shared by every archive dir**, so one login covers all chats. `migrate_session` relocates a pre-existing `/session.session` to the shared path on first run rather than forcing a re-login; it is a no-op once the shared session exists. It uses `shutil.move`, not `os.replace`, because the two paths can be on different filesystems. -- cgit v1.2.3