Open-Vocabulary Gaze Object Prediction: Benchmark and Method

2026-07-21

AI Summary

This work addresses the limited generalizability of existing gaze target prediction methods, which are constrained by closed-vocabulary assumptions and scene-specific training data, rendering them ineffective for long-tail or unseen categories in real-world settings. To overcome this, the study introduces the first open-vocabulary gaze target prediction task, establishes DiSG—a new benchmark comprising 86 in-the-wild categories—and proposes a novel framework featuring text-driven object discovery and gaze-guided selection modules. Furthermore, it incorporates GIST (Gradient-Informed Selective Tuning), a parameter-efficient fine-tuning strategy leveraging gradient information to enhance semantic generalization. The proposed approach significantly outperforms current state-of-the-art models under both open- and closed-vocabulary evaluation protocols, demonstrating its robustness and adaptability in complex, real-world scenarios.

Abstract

Gaze Object Prediction (GOP) aims to localize and recognize the objects humans attend to, a task crucial for understanding human-centric interactions. However, existing methods are typically trained under a closed-vocabulary paradigm with a fixed label space and evaluated on scene-specific datasets, limiting their applicability to real-world scenarios where gaze targets often follow a long-tail distribution or belong to unseen categories. To address this gap, we introduce Diverse Scenes for Gaze object prediction (DiSG), a benchmark containing 86 in-the-wild categories that facilitates the evaluation of Open-Vocabulary GOP (OVGOP). Building on DiSG, we propose a framework that leverages text-driven object discovery to localize potential gaze candidates, with a gaze-guided selection module to pinpoint the intended target from the candidate objects. Furthermore, to better capture semantic knowledge across diverse in-the-wild categories, we introduce Gradient-Informed Selection Tuning (GIST) to selectively update parameters most relevant to a given class vocabulary. Extensive experiments demonstrate that our proposed model performs effectively in open-vocabulary settings and also outperforms existing methods in the conventional closed-vocabulary setting. The benchmark and code is available at https://github.com/sensniu/ovgop.

Problem

• Gaze Object Prediction
• Open-Vocabulary
• Long-tail Distribution
• Unseen Categories
• Human-Centric Interaction

Innovation

• Open-Vocabulary Gaze Object Prediction
• DiSG benchmark
• text-driven object discovery
• gaze-guided selection
• Gradient-Informed Selection Tuning