GUI-Lens: Coarse-to-Fine Cropping for GUI Grounding with General-Purpose VLMs

2026-08-04

AI Summary

This work addresses the challenge that general-purpose vision-language models (VLMs) struggle to accurately localize interactive GUI elements specified by natural language instructions in high-resolution, element-dense graphical user interfaces. To overcome this limitation, the authors propose a coarse-to-fine iterative localization framework that decomposes the task into sequential steps of region cropping, zooming, and verification through an active visual observation mechanism. By integrating OCR-extracted text, UI component detection, and a coordinate reference system, the framework guides any off-the-shelf VLM to progressively focus on the target region and retrospectively map the precise click location. This approach mitigates error propagation inherent in single-step prediction and operates in a plug-and-play manner. Experiments across four GUI benchmarks and three VLM backends demonstrate performance gains of up to 24.9 percentage points in overall accuracy, achieving state-of-the-art results with GPT-5.5.

Abstract

GUI grounding maps natural-language instructions to click locations and is essential for reliable GUI agents. The task remains difficult on high-resolution, densely populated interfaces because a vision-language model (VLM) may recognize a requested control without locating it precisely enough for interaction. Most existing methods provide various forms of localization assistance, but still rely on a direct click prediction, allowing visual ambiguity or an inaccurate initial estimate to propagate to the final result. In this paper, we introduce GUI-Lens, a coarse-to-fine grounding framework that allows a general-purpose VLM to determine the target through active visual observations. Specifically, GUI-Lens extracts OCR text and detected UI components from the screenshot and presents their positions as coordinate references. Using the instruction, the current view, and these references, the VLM selects the region and scale of the next view, which is cropped and enlarged to provide finer visual details. This process continues over successively focused views until the target is determined. Proposed crops and clicks are checked against the instruction throughout the process, and the final local position is mapped back to the original screen coordinates. Experiments on four GUI grounding benchmarks and three general-purpose VLM backends show that GUI-Lens improves overall grounding accuracy by up to 24.9 percentage points and achieves state-of-the-art performance with GPT-5.5.

Problem

• GUI grounding
• vision-language models
• click localization
• high-resolution interfaces
• visual ambiguity

Innovation

• coarse-to-fine cropping
• GUI grounding
• vision-language models
• active visual observation
• UI component localization