Design the Image the Model Sees, Not Just the Prompt
I shoot a lot of RAW photos and keep far too many of them. So I built a small tool that asks a vision model to triage each shoot: keep the RAW, convert it to a JPEG, or shrink it to a small copy. I expected the work to be prompt tuning. It wasn't. Almost all of the effort, and almost all of the improvement, came from deciding what image the model actually gets to look at.
This post is about that idea: when a model judges images, the image you hand it is the real interface. Design it like one.
The obvious approach, and why it fails
My first version did what everyone does. Export a preview of each photo, attach it, and ask: "Is this sharp? Is it worth keeping?" The answers sounded confident and were mostly noise.
The reason is simple once you see it. Vision models resize large images before they look at them. A 24-megapixel frame gets shrunk to something like a megapixel and a half. At that size, a photo where the eyes are pin-sharp and a photo where focus landed on the ear look identical. The detail that decides the verdict has been averaged away before the model sees a single pixel. No wording in the prompt can bring it back.
The second problem: I was asking the model to measure things. "Is this sharp?" is a measurement question, and models are bad at measuring. Code is good at it.
The idea: build a composite for the model
Instead of sending the photo, I send a purpose-built composite. One row per photo, three panels side by side, with a strip of measured numbers above them:
- The full frame, scaled down, with two boxes drawn on it showing where the other panels come from. This is for composition and context.
- A medium zoom around the focus point, covering about a quarter of the frame's width. This shows the subject: expression, eyes, gesture.
- A 100% crop at the focus point: real sensor pixels, not resampled. This is the only panel where sharpness can actually be judged.
- A label strip with numbers code has already measured: exposure, a sharpness score, where the focus point came from, which burst the photo belongs to, and warnings such as a shutter speed too slow for the focal length or clipped highlights.
Then the part that makes it work: I size each sheet so the model never downsamples it. Three rows per sheet, each panel 400 pixels wide, keeps the whole sheet a little over a megapixel, just under the size where the model would start resizing. The 100% crop stays at 100%. What I lay out is exactly what the model sees.
Here is the core of it as a Pillow sketch:
from PIL import Image, ImageDraw
PANEL = (400, 267)
def fit(img, size):
img = img.copy()
img.thumbnail(size)
panel = Image.new("RGB", size)
panel.paste(img, ((size[0] - img.width) // 2, (size[1] - img.height) // 2))
return panel
def build_row(photo, focus_xy, label):
fx, fy = focus_xy # pixel coordinates in the full image
w, h = photo.size
mw, mh = w // 4, int(w // 4 * 2 / 3) # medium zoom: a quarter of the width
mid = photo.crop((fx - mw // 2, fy - mh // 2, fx + mw // 2, fy + mh // 2))
one = photo.crop((fx - PANEL[0] // 2, fy - PANEL[1] // 2,
fx + PANEL[0] // 2, fy + PANEL[1] // 2)) # no resize: true pixels
row = Image.new("RGB", (3 * PANEL[0] + 12, PANEL[1] + 26), (24, 24, 24))
ImageDraw.Draw(row).text((4, 5), label, fill=(235, 235, 235))
for i, panel in enumerate([fit(photo, PANEL), fit(mid, PANEL), one]):
row.paste(panel, (i * (PANEL[0] + 6), 26))
return rowWhere does "the focus point" come from?
Both zoom panels depend on picking the right spot. Pick wrong and the 100% crop shows a sharp patch of background while the face is soft, and the model will happily call the photo sharp. So the focus point comes from a fallback chain, most trustworthy first:
- The camera's own autofocus point, read from the file's metadata. Most cameras record where they focused, each in their own format. If it is there, it is the ground truth for "where the photographer meant to be sharp".
- The largest detected face. For people pictures, the face is almost always what matters.
- The sharpest region of the image, found by sliding a tile across the frame.
def focus_point(photo, meta):
if (p := af_point_from_metadata(meta)):
return p, "camera AF"
if (p := largest_face_center(photo)):
return p, "face"
return sharpest_tile_center(photo), "sharpest"The important detail is that the source goes onto the label too, because it changes what the crop means. If the source is the camera's AF point and the crop is soft, focus was missed. If the source is "sharpest" and the crop is soft, the sharpest part of the image is soft, so the whole frame is blurred. Same picture, different conclusion. I tell the model this in its instructions, and it uses it.
Compare within a burst, never against a threshold
My sharpness score is the variance of the Laplacian (a cheap edge-strength measure), taken at the focus crop and at the sharpest tile, plus the ratio between the two. It is useful, but the raw number means almost nothing on its own. A textured subject at low ISO scores high; a smooth face at high ISO scores low, even when both are perfectly in focus.
So there is no "sharp if above 200" rule anywhere. Instead, code groups photos into bursts (shots taken a second or two apart that look nearly the same), and the instructions say to compare scores only within a burst or between similar shots. Among five frames of the same moment, the one with clearly higher focus sharpness is the keeper, and that comparison is reliable even when the absolute value isn't. The ratio of focus to peak sharpness is a second hint: when the camera's AF point is much softer than the sharpest spot in the frame, focus probably landed somewhere else.
The model proposes, a human decides
Every verdict comes back as structured data: the decision, a confidence of high, medium or low, and one sentence naming what the model saw ("eyes soft at 100%; the next frame in the burst is sharp"). The instructions say to use low whenever unsure, and the review page has a "low confidence" filter. In practice I review the low ones carefully and skim the rest.
Nothing touches the photos until I confirm a set in that page. Applying a confirmed set moves originals to a trash folder instead of deleting them, writes a log of every change, and an undo command reverses the last run. A model that is wrong one time in twenty is fine when every mistake costs a keypress to fix.
Running it as a job
The model runs headlessly, one job per set of up to a hundred photos, from a small local page with a queue. Each job runs the Claude CLI in print mode with JSON output and tools restricted to reading files and writing the verdict file. It never gets a shell, and it never sees the original photos' folder with write access. The JSON output includes the run's cost, which I record per job and show in the jobs panel, so I always know what a shoot cost to triage before I queue the next one.
Trade-offs
- It is more code than a prompt. Metadata parsing alone has a branch per camera brand. If your images don't hide their important detail at small sizes, a plain resized image may be enough.
- Composites cost tokens. A full-resolution sheet uses more input than a thumbnail. I trade that for verdicts I can trust, and the per-run cost makes the trade visible.
- Measured numbers can mislead. A printed score looks authoritative. I had to tell the model explicitly that intentional softness (panning, fog, shallow depth of field) is not a defect, and that numbers are for comparison, not judgement.
Takeaway
If a vision model's answers are vague, look at what it actually receives before rewriting the prompt. Crop to what matters, keep it at true scale, print the measurements code is better at, and leave the model the one job it is good at: looking and comparing.