Problem
Research questions and friction points this paper is trying to address.
image-space reasoning
visual rule discovery
image editing
worksheet benchmark
end-to-end problem solving
Image-Space Rule Discovery
🤖 AI Summary
This work investigates whether image editing models can autonomously discover visual rules in pixel space and perform human-like worksheet tasks—such as IQ tests—end-to-end, encompassing instruction comprehension, answer reasoning, and precise image editing. To this end, we introduce WISRD, the first systematic benchmark comprising 11 core tasks and four categories of reasoning stress tests, along with an automated, stringent scoring agent for objective model evaluation. Experiments demonstrate that state-of-the-art image editing models, notably Nano Banana Pro, can partially achieve complex visual reasoning without external prompting: Nano Banana Pro attains a 48.7% pass rate on the reference-free V0–V3 subset, and achieves 70.0% and 22.9% accuracy on 4×4 Sudoku and RAVEN tasks, respectively, substantially outperforming existing approaches.
📝 Abstract
Can image-editing models discover visual rules in image space and complete problem-solving end-to-end? We tackle this question in the spirit of a human worksheet test (e.g., an IQ test), using problems that require models to read image-based instructions, recognize the problem, infer the answer, bind it to the correct destination, control output count, suppress unnecessary edits, and preserve the input and format. We introduce WISRD, a Worksheet Image-Space Rule Discovery benchmark with 11 core tasks under eight information conditions, spanning localized marking, filling, copying, counting, and no-edit suppression, together with four supplementary reasoning-stress probes for multi-step spatial manipulation, abstract pattern reasoning, logical inference, and constraint-based problem solving. We identify three key findings as follows. (i) Among the frontier image-editing models evaluated, Nano Banana Pro achieves the highest score. On the shared V0--V3 no-reference subset, the Auto-Strict proxy pass rates are 48.7% for Nano Banana Pro, 13.4\% for Qwen-Image-Edit, 11.5% for FLUX.2 Klein 4B API, 11.3% for FLUX.2 Klein 4B open-weight, and 0.0% for InstructPix2Pix. (ii) Analysis reveals that current image-editing models can partially rely on rendered in-image instructions even when the external prompt is absent or merely generic. (iii) In small supplementary diagnostics, Nano Banana Pro achieves 70.0% on 4-by-4 Sudoku and 22.9% on public RAVEN pattern-discovery items in image space.
Innovation
Methods, ideas, or system contributions that make the work stand out.
image-space rule discovery
worksheet-based reasoning
visual instruction following
image editing benchmark
in-image reasoning
🔎 Similar Papers
💼 Related Jobs
