Why MLLMs Struggle to Count: Overcoming Individuation and Aggregation Bottlenecks with ConvStack

2026-09-29

AI Summary

This study addresses the limitation of multimodal large language models in fine-grained counting tasks, which stems from visual patching and attention-based compression mechanisms. We formally define the "individuation-aggregation" dual bottleneck for the first time and propose ConvStack, a lightweight architecture that injects local structural information into the visual token space via stacked local convolutions while enabling explicit aggregation through zero-initialized residual connections. This approach operates without modifying the backbone network, significantly improving dense object counting accuracy and spatial understanding capabilities while preserving the model's general visual performance.

Abstract

Multimodal Large Language Models (MLLMs) consistently struggle with fine-grained visual counting, yet the underlying causes remain poorly understood. In this work, we present a mechanistic analysis of this failure mode, identifying two critical bottlenecks inherent to the global attention pipeline of MLLMs. First, we reveal an individuation bottleneck stemming from image patchification: because Vision Transformers process patches independently, they struggle to group fragmented geometric features across boundaries into distinct object representations. Second, we identify a collapse in the subsequent counting aggregation process, where representation separation rapidly diminishes as numerosity increases due to attention compression. Identifying and formalizing these twin bottlenecks constitutes our first major contribution. To overcome them, we propose ConvStack, a lightweight architecture that operates directly in the visual token space to explicitly aggregate and inject local spatial structures via zero-initialized residual connections. By explicitly addressing the individuation bottleneck, ConvStack provides unambiguous geometric evidence for downstream aggregation. Remarkably, by fine-tuning exclusively on counting tasks, the model achieves substantial improvements in dense object counting and broader spatial understanding benchmarks, without compromising on general visual capabilities.

Problem

• Multimodal Large Language Models
• Visual Counting
• Individuation Bottleneck
• Aggregation Bottleneck
• Vision Transformers

Innovation

• Multimodal Large Language Models
• Visual Counting
• ConvStack
• Individuation Bottleneck
• Attention Compression