TargetFinder: Detecting Widgets from Pixels on Desktop Interfaces

2026-07-22

AI Summary

Existing target-aware pointing techniques struggle to achieve broad deployment in desktop environments due to the lack of application-agnostic geometric information about interface controls. This work proposes TargetFinder, a computer vision–based real-time desktop control detection system that, for the first time, enables application-independent, high-precision control recognition. By constructing a large-scale annotated dataset spanning Windows, macOS, Ubuntu, and web applications, and integrating a fine-tuned lightweight YOLO model with a low-latency screen monitoring mechanism, TargetFinder substantially advances the state of the art. Experimental results demonstrate that TargetFinder outperforms OmniParser and REMAUI in detection accuracy and has been successfully integrated with Bubble Cursor and Semantic Pointing, thereby validating the feasibility and practicality of system-level target-aware interaction.

Abstract

''Target-aware'' pointing techniques, like Bubble Cursor or Semantic Pointing, outperform traditional pointing by leveraging knowledge of target locations. Yet the lack of application-agnostic widget geometry information limits their adoption across the desktop. We present TargetFinder, a computer vision-based system for real-time detection of GUI widgets. TargetFinder leverages several fine-tuned YOLO networks trained on a new dataset of 520 annotated desktop screenshots (~38,000 annotations) spanning Windows, macOS, Ubuntu, and web interfaces. TargetFinder uses lightweight screen monitoring and low-latency detection, achieving millisecond responsiveness suitable for interactive use. Evaluations show that TargetFinder outperforms the baseline methods (OmniParser and REMAUI), while system-wide implementations of Bubble Cursor and Semantic Pointing demonstrate the feasibility of deploying universal target-aware techniques that work across applications. We release the dataset, models, annotation tool, and an open-source library for research and applications.

Problem

• target-aware pointing
• GUI widget detection
• desktop interfaces
• application-agnostic

Innovation

• Target-aware pointing
• GUI widget detection
• YOLO
• cross-platform desktop interfaces
• real-time computer vision