JigShape: Evaluating Visual-Geometric Reasoning in VLMs through Jigsaw Puzzles

2026-07-30

AI Summary

This work addresses a critical limitation in current vision-language models (VLMs)—their poor capacity to jointly reason with visual content and geometric constraints. To this end, we introduce the first scalable jigsaw benchmark based on mortise-and-tenon interlocking structures, which enforces strong geometric constraints alongside visual cues to provide unambiguous ground truth. We systematically evaluate state-of-the-art VLMs across four grid densities (4×4 to 16×16) using 95K instances under both zero-shot and fine-tuned settings. Results reveal that, except for GPT-5.5 slightly surpassing random chance on 4×4 zero-shot puzzles, all models perform near-randomly. Although fine-tuning achieves 97% accuracy on 4×4 puzzles, performance collapses to below 5% for grids larger than 12×12, exposing a “scaling cliff” in VLMs’ ability to handle complex visual-geometric reasoning.

Abstract

Jigsaw puzzle solving requires jointly reasoning about visual content and geometric constraints, yet existing benchmarks use rectangular cuts that create ambiguous ground truth in texture-repeated regions. We introduce \textit{\ours{}}, a benchmark with tab-and-blank interlocking pieces where geometric constraints provide strong local compatibility requirements that, combined with visual content, yield unambiguous ground truth. Across 95K instances at four grid densities (4$\times$4 to 16$\times$16), we find that \textbf{zero-shot VLMs largely lack geometric reasoning}: only one of five frontier models (GPT-5.5) exceeds random baseline on 4$\times$4 puzzles, while all others perform at chance level. While supervised fine-tuning achieves $>$97\% on 4$\times$4, \textbf{all models collapse on larger grids}: GPT-5.5 drops from 70\% to near-random on 8$\times$8, and even fine-tuned models fall below 5\% on 12$\times$12. This ``scaling cliff'' suggests current architectures cannot maintain consistent constraint satisfaction as the number of pieces increases. \ours{} establishes scalable geometric reasoning as an open challenge for vision-language models.

Problem

• visual-geometric reasoning
• jigsaw puzzles
• vision-language models
• geometric constraints
• scalable reasoning

Innovation

• geometric reasoning
• jigsaw puzzles
• vision-language models
• benchmark
• scaling cliff