Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents

2024-10-07

arXiv.org

AI Summary

GUI agents have long relied on noisy, incomplete textual representations—such as HTML or accessibility trees—hindering human-like visual perception and pixel-precise interaction. This work introduces UGround, the first purely vision-based, cross-platform GUI element localization model that directly regresses element coordinates from raw screen pixels. Methodologically: (1) we construct the largest GUI visual localization dataset to date (1.3M screenshots, 10M annotated elements); (2) we propose an efficient training paradigm combining web-synthesized data with lightweight LLaVA adaptation; and (3) we introduce multi-scale visual feature alignment coupled with coordinate regression. Evaluated across six benchmarks, UGround achieves up to a 20% improvement in visual localization accuracy. Remarkably, GUI agents powered solely by UGround’s visual input outperform state-of-the-art multimodal (text-augmented) approaches—demonstrating that high-fidelity visual grounding alone suffices for robust GUI interaction.

Technology Category

• Computer Vision: Large Vision Models
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Humans and AI: Intelligent User Interfaces

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Multimodal large language models (MLLMs) are transforming the capabilities of graphical user interface (GUI) agents, facilitating their transition from controlled simulations to complex, real-world applications across various platforms. However, the effectiveness of these agents hinges on the robustness of their grounding capability. Current GUI agents predominantly utilize text-based representations such as HTML or accessibility trees, which, despite their utility, often introduce noise, incompleteness, and increased computational overhead. In this paper, we advocate a human-like embodiment for GUI agents that perceive the environment entirely visually and directly perform pixel-level operations on the GUI. The key is visual grounding models that can accurately map diverse referring expressions of GUI elements to their coordinates on the GUI across different platforms. We show that a simple recipe, which includes web-based synthetic data and slight adaptation of the LLaVA architecture, is surprisingly effective for training such visual grounding models. We collect the largest dataset for GUI visual grounding so far, containing 10M GUI elements and their referring expressions over 1.3M screenshots, and use it to train UGround, a strong universal visual grounding model for GUI agents. Empirical results on six benchmarks spanning three categories (grounding, offline agent, and online agent) show that 1) UGround substantially outperforms existing visual grounding models for GUI agents, by up to 20% absolute, and 2) agents with UGround outperform state-of-the-art agents, despite the fact that existing agents use additional text-based input while ours only uses visual perception. These results provide strong support for the feasibility and promises of GUI agents that navigate the digital world as humans do.

Problem

• Enhancing GUI agents' visual grounding for real-world applications.
• Reducing noise and computational overhead in GUI element recognition.
• Developing a universal visual grounding model for diverse platforms.

Innovation

• Visual grounding models for GUI agents
• Pixel-level operations on GUI elements
• Web-based synthetic data for training