GEB-Bench: Abstract Structures Told in Many Voices

2026-08-04

AI Summary

This work proposes GEB-Bench, the first systematic benchmark for evaluating large models’ capacity to transfer abstract structural patterns across modalities, with a focus on deep structural concepts such as self-reference and strange loops across natural scenes, folktales, mathematical theorems, and program skeletons. Leveraging multimodal data (images, text, mathematics, and code), formal structural annotations, and a generative evaluation pipeline, the benchmark isolates structural consistency from superficial cues. Evaluation of twelve state-of-the-art models reveals that none effectively transfers abstract structures across modalities; even the most advanced models only partially narrow the performance gap. Error patterns are highly consistent across models, heavily influenced by surface-level complexity, and increased model capacity merely enhances robustness rather than fundamental structural reasoning ability. This study thus exposes a fundamental limitation in current models’ structural cognition.

Abstract

Can a model look at a river delta and a lightning bolt and see that they share a structure? We introduce GEB-Bench, a benchmark whose unit is an abstract structural motif--self-reference, a strange loop, a Mobius twist--in the spirit of Godel, Escher, Bach. Each motif is told in several voices: a natural scene whose composition is the structure, a folk story whose telling enacts it through a mechanically checkable form device, a mathematical theorem, and a programmatic skeleton; surface parameters are declared nuisance variables and never scored. Motifs, voices, and the structural changes between them form a small cross-modal category, and GEB-Bench's tasks are its questions. Evaluating twelve open and proprietary models, we find that abstraction failure is lawful. The central finding is a gap between recognition and cross-voice mapping: models identify a structure within one voice far better than they carry it across voices; every model pays this tax, and mapping strong enough to narrow it appears only at the frontier tier. Two patterns support it. Errors align more strongly with the designed formal geometry than with measured perceptual geometries, and frontier models from different vendors converge on the same wrong answers; and surface complexity taxes every model that reads structure, with capacity buying headroom rather than immunity. GEB-Bench is fully generative and released with its pipeline.

Problem

• abstract structure
• cross-modal mapping
• structural motif
• abstraction failure
• GEB-Bench

Innovation

• abstract structural motifs
• cross-modal mapping
• GEB-Bench
• structural generalization
• generative benchmark