PerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Models

2026-07-27

AI Summary

Current evaluations struggle to disentangle visual perception errors from reasoning or knowledge deficiencies in multimodal large language models. This work proposes a bottom-up approach that systematically identifies and isolates ten atomic visual perception capabilities by analyzing early failure cases of 16 state-of-the-art models across 42 benchmarks. The authors construct a curated dataset of 3,000 human-verified questions with concise answers, enabling precise assessment of pure perceptual abilities. Experimental results reveal that all models achieve less than 60% accuracy on these atomic perception tasks, exhibit the weakest hallucination rates in perception-related scenarios, and—despite similar overall scores—demonstrate markedly distinct capability profiles. This study establishes the first evaluation benchmark dedicated exclusively to atomic visual perception.

Abstract

We introduce PerceptionBench, a benchmark specifically designed to evaluate the atomic visual perception capabilities of Multimodal Large Language Models (MLLMs). Existing benchmarks often fail to isolate perception: holistic evaluations conflate perceptual errors with failures in reasoning or domain knowledge, while application-driven benchmarks only cover narrow, fragmented domains shaped by heuristic designs. To address these limitations, PerceptionBench adopts a bottom-up approach: by diagnosing the earliest failure points in the responses of frontier MLLMs across 42 existing benchmarks, we construct an error taxonomy whose perception branch defines ten atomic perceptual capabilities. Guided by this taxonomy, we construct 3,000 verified questions with short, unambiguous answers, each isolating a single capability, with difficulty stemming from perception rather than reasoning or knowledge. Benchmark results across sixteen frontier MLLMs reveal that atomic perception remains largely unsolved---no model reaches 60\% accuracy, perception-related hallucination is the weakest capability on average, and similar overall scores conceal sharply divergent capability profiles. PerceptionBench thus provides a capability-level standard for measuring and diagnosing the visual perception boundaries of MLLMs.

Problem

• atomic visual perception
• multimodal large language models
• perception benchmark
• perceptual errors
• capability evaluation

Innovation

• atomic visual perception
• multimodal large language models
• perception benchmark
• error taxonomy
• capability-level evaluation