Published benchmark
Q3 2026 edition
The best AI judge we tested agrees with a working product photographer 96% of the time. Most of the models people rank as the best at vision agree far less, and the current number one on the largest public vision leaderboard missed 5 of the 22 defects a professional caught. That gap is the reason we built this.
When you generate a commercial image of a real product, the product in the picture has to be the actual product. The temples of the glasses, the hinge, the lens shape, the bridge, the material and color. If any of that drifts, you have shipped a wrong product image to a client, and no amount of nice lighting saves it.
The Product Fidelity Bench asks one question: can an AI model look at the real product photo, look at an AI-generated image using that product, and correctly say whether they match? We judge the match overall and per region. AI vision can assist as a judge in determining if the product was correctly represented. This benchmark measures which models can actually do that job.
The headline metric is the false-pass rate: the share of defective images a judge approves. In plain terms, a false pass is the judge saying "looks good" about an image that is wrong, which is the exact failure that reaches a client and costs you the account. We also track agreement with the professional, false-fails (good work the judge wrongly rejects, which wastes money by throwing away usable images), region-level false alarms, latency, and cost per image.
The largest public vision leaderboard is LMArena's Vision Arena. It collects crowd preference votes on conversational answers about images: captioning, reading text in a photo, homework help, jokes. There is no category for comparing an image to a reference or judging whether it is faithful. Preference voting also rewards the agreeable, confident answer, which is the opposite of what a strict judge does. A good judge has to be willing to say no.
So the skill that leaderboards rank, conversational visual assistance, is not the skill a QA judge needs. Nobody was measuring whether AI can say no to a bad product image. There was no adversarial vision eval benchmark for this, so we decided to create our own.
The gap shows up in the numbers. The current number one model on Vision Arena, Claude Fable 5, missed 64% of the defects the professional caught under our first rubric, and still misses 23% (5 of 22) under the corrected one. Arena rank does not transfer to commercial QA judgment.
Scenario 1 (Q3 2026 edition): eyewear. Clear acetate frames with fine metal details, which are hard to render and easy to get subtly wrong. 24 AI-generated images. A working AI-photography professional judged each one against the real product using a real client-acceptance bar: 22 rejected, 2 accepted. That professional judgment is the ground truth. Your eyeball was the eval; now we are bottling it.
Each model saw the reference product photo plus a generated image and returned a verdict, overall and per region (temples, hinge, lens shape, bridge, material and color). We score every model against the professional's labels.
Rows marked "+ crop" ran a two-stage protocol. The same model is first asked to locate the product in the generated image and return a bounding box. Locating is a much easier vision task than fine judging, so even models that fail at fidelity can find the glasses on a face. We expand that box by 25% on every side as insurance, then crop at the image's native resolution. No upscaling and no resampling: upscaling adds no information, and AI super-resolution would hallucinate detail, which is the one thing a fidelity judge must never look at. The zoom effect comes from how vision APIs work. They shrink every image to a fixed pixel budget, so a full portrait spends only a fraction of that budget on the product, while the crop spends all of it there. An effective 2 to 4x zoom without a single invented pixel. Each model crops for itself, so a bad localization compounds into a bad judgment. That is deliberate: we measure the whole pipeline, and every crop is saved so verdicts can be audited against what the judge actually saw.
One honest detail about how we got here. Our first rubric, v0.1, asked judges to verify text on the temples that did not actually exist on the reference product. Half of what first looked like model blindness was our own rubric asking the wrong question. We corrected it to v2, re-ran everything against clean labels, and the correction lifted every model. We are telling you this on purpose, because benchmark quality gates everything downstream, which is why we publish the methodology and the limitations instead of only the scores.
Leaderboard
Ranked by agreement with the professional, on 24 images with corrected labels.
96%
Top agreement with professional
| Rank | Model | Agreement | Defects missed (of 22) | False-fails | Latency/image | Cost/image |
|---|---|---|---|---|---|---|
| 1 | GPT-5.5-pro | 96% | 1 | 0 | 109s | $0.58 |
| 2 | GPT-5.4-mini + crop | 92% | 0 | 2 | 14.1s | $0.008 |
| 3 | GPT-5.5 | 92% | 1 | 1 | 34s | $0.03 |
| 4 | GPT-5.4-mini | 88% | 1 | 2 | 9.5s | $0.005 |
| 5 | Gemini 3.1 Pro + crop | 83% | 3 | 1 | 19s | $0.02 |
| 6 | Gemini 3.5 Flash + crop | 83% | 2 | 2 | 6.4s | $0.007 |
| 7 | GPT-5.5 + crop | 83% | 3 | 1 | 42s | $0.06 |
| 8 | GPT-5.5 low effort | 83% | 3 | 1 | 16s | $0.05 |
| 9 | GPT-5.5 low effort + crop | 79% | 5 | 0 | 25s | $0.09 |
| 10 | Claude Fable 5 | 79% | 5 | 0 | 13.5s | $0.11 |
| 11 | Gemini 3.1 Pro | 79% | 5 | 0 | 14s | $0.02 |
| 12 | Claude Fable 5 + crop | 67% | 7 | 1 | 22s | $0.21 |
| 13 | Claude Sonnet 5 + crop | 58% | 10 | 0 | 9.2s | $0.04 |
| 14 | Claude Sonnet 5 | 54% | 11 | 0 | 6.2s | $0.03 |
| 15 | Claude Opus 4.7 + crop | 46% | 13 | 0 | 11.3s | $0.09 |
| 16 | Claude Haiku 4.5 + crop | 42% | 14 | 0 | 6.6s | $0.006 |
| 17 | Gemini 3.5 Flash | 42% | 14 | 0 | 4.6s | $0.005 |
| 18 | Claude Opus 4.7 | 38% | 15 | 0 | 6.9s | $0.048 |
GPT-5.4-mini with crop missed zero of the 22 defects. Its two false-fails mean it sometimes rejects good work, which a second-tier check can recover.
Gemini 3.1 Pro + crop had the cleanest region attribution, with 5% false alarms at the region level.
01
The best judge agrees with the professional 96% of the time. That is good enough to screen work at scale. None of them replace the professional eye yet, and most of the field is well below that top number.
02
It helped some models and hurt others, so you have to measure it. Cropping to the product lifted Gemini 3.5 Flash by 42 points and nudged most of the mid-field up a little. The mechanism is simple: vision APIs shrink your image down to a fixed pixel budget, so on a full scene the product gets a few pixels, but cropping spends the whole budget on the product. But cropping reduced accuracy for the strongest full-image judges. GPT-5.5 dropped 8 points and Claude Fable 5 dropped 12, because those models were using whole-scene context like the scale of the product on the face and its proportions, which the crop throws away. The lesson: whether zooming helps is a per-model property you have to measure, which is exactly what a benchmark is for.
03
Being trained to be helpful and agreeable pulls a model in the opposite direction from strict judging. The most agreeable models are not the best judges.
04
The same GPT-5.5 family judged wildly differently through an agent command-line harness versus a clean API call, from near-blanket rejection to calibrated verdicts. The plumbing around the model matters as much as the model.
05
GPT-5.4-mini with crop caught all 22 defects, missing zero, in 14 seconds at under a cent per image. That makes the two-tier production setup concrete: this cheap fast model screens everything, and a stronger model reviews only its rejections, so nothing bad slips through and you spend the expensive model only on the close calls.
06
The v0.1 error that asked models to verify text that did not exist cost every model accuracy until we fixed it. If we can get the question wrong, so can anyone publishing scores without showing their method.
We would rather state these than have you find them.
The Product Fidelity Bench updates once per quarter. Each edition adds a scenario chosen because it is genuinely hard to get right. Next up: game tiles with engraved characters (mahjong), and hotel-grade fabric texture (percale bedsheets). Rankings will move as models ship and as the scenario set grows, and every edition will carry its own version number and its own labels.
We run AI product photography for real clients, which means every image we send has to pass a real acceptance bar. The Product Fidelity Bench is not a marketing exercise. It is how we choose the models that check our own work before it reaches a client. We are publishing it because the question it answers, whether AI can tell a right product image from a wrong one, matters to anyone buying AI photography, and because there was no honest answer available anywhere else.
Methodology questions or model submissions: contact the Dezygn team.
