Why VLMs Miss Small Objects, and When Zooming In Is Safe

2026-10-06

AI Summary

This study addresses the tendency of Vision-Language Models (VLMs) to overlook small objects in large images and the absence of theoretical safety guidance for local magnification strategies. We construct a theoretical framework based on visual token density and content coverage, introducing the first recall-cost frontier theory for small object recognition in VLMs. This framework rigorously defines the safety boundaries of local magnification and proves that image decomposition does not degrade recall under specific conditions. Large-scale empirical evaluations validate these theoretical predictions, demonstrating that the proposed strategy improves recall by up to 0.28 on architectural drawings without significant performance degradation. The associated code and datasets have been made publicly available.

Abstract

Vision-language models (VLMs) often miss small objects in large images. We ask three questions: what limits them, which of these limits better models can remove, and whether the classical way of handling large images, local decomposition, still has a future. We answer them with a theory built on two quantities of the image interface: S, the number of visual tokens across an object's side, and L, the content a call must cover. The limits: a W x H image sent whole within N tokens gives an object of side m at most m*sqrt(N/(WH)) tokens per side. Doubling the token budget N therefore raises the largest S a whole image can reach by only 41%, and seeing an image at the S an object needs costs at least order S^2 tokens whatever the model. If recognition improves gradually with S, any search strategy, zoom agents included, obeys a recall-cost frontier. What better models can change: the S an object needs and how much one call can carry, measured in bits per object found; the coverage cost remains. Decomposition: yes. Assuming only that more tokens per object and less content per call do not hurt on average, splitting an image cannot lower recall if no view zooms out relative to the whole image and views overlap by one object. Neither part of this condition can be dropped; we bound the cost of every such decomposition, and a simple rule approaches the bound as the image grows. We test the theory in about 177,000 requests on 797 images. On controlled images, none of 40 orderings predicted for 8 VLMs is violated. Checked after the fact on drawings, floor plans, natural and synthetic images, 61 of 89 implied orderings hold significantly and 4 fail, all for OpenAI models given more pixels than their default path. On construction drawings the rule never significantly lowered recall relative to the whole image and raised it by up to 0.28. Code and data: https://github.com/shijunzhe/vlm-small-objects

Problem

• Vision-language models
• small object detection
• visual tokens
• image decomposition
• large images

Innovation

• Vision-Language Models
• Small Object Detection
• Visual Tokens
• Image Decomposition
• Recall-Cost Frontier