Add TODO with img2img (image editing) idea for future brainstorming

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

1 files changed, 38 insertions, 0 deletions

diff --git a/TODO.md b/TODO.md
new file mode 100644
index 0000000..a1e4eb8
--- /dev/null
+++ b/TODO.md
@@ -0,0 +1,38 @@
+# TODO / future ideas
+
+Ideas for later, to brainstorm and spec before building. Nothing here is committed
+to scope yet. Each entry should go through the brainstorming flow (like the model
+menu + daemon work did) before implementation.
+
+## Image editing (img2img)
+
+Pass an existing image plus a prompt describing the edits, get a modified image back.
+
+**Why it fits:** This is SDXL img2img. Same base model, same fp16-fix VAE, same 12GB
+budget. `AutoPipelineForImage2Image.from_pipe(txt2img_pipe)` derives the img2img pipeline
+from the already-resident model, so no second load and no extra VRAM. Still SDXL-arch,
+single image, no refiner/Flux/batching, so it stays within the project's bounded scope.
+
+**Rough shape (to be confirmed in brainstorming):**
+- `generate.py`: a `generate_img2img(pipe, image, prompt, ..., strength)`; load the input
+  via PIL, resize to a multiple-of-8 dimension. `build_pipe` unchanged; derive the img2img
+  pipe from it.
+- `server.py`: a new `/edit` endpoint. Image transport likely base64-in-JSON (simple,
+  stdlib-only, fits the jq-free / no-new-dep spirit; ~33% size bloat is fine for one local
+  image). Reuses the resident model.
+- `imggen` wrapper: either a subcommand `imggen edit input.png "make it snowy" -o out.png
+  [--strength 0.6]`, or an `-i/--input` flag on the normal prompt path. Subcommand is
+  probably cleaner since start/stop/status/list already use subcommands.
+- Registry unchanged: all three models (sdxl/realvis/pony) can do img2img.
+- `strength` is the key new knob: 0 = ignore prompt (return input), 1 = ignore input
+  (pure txt2img); ~0.5-0.75 typical for edits. Leave it exposed for tuning.
+
+**Open questions to resolve first:**
+1. Interface: `edit` subcommand vs `-i/--input` flag.
+2. Image transport to the daemon: base64-in-JSON vs multipart.
+3. Inpainting (mask a region to edit) is a separate third pipeline
+   (`XLInpaintPipeline` + a mask input), more surface. Start with whole-image img2img;
+   add masked inpainting later only if wanted.
+
+**VRAM:** img2img peak is roughly the same as txt2img (same UNet passes, often fewer steps
+since it starts from a noised input), so it fits 12GB. Verify on the real Arc during build.