SaaF: Scene-Specific Ambiguity-Aware 3D Language Fields towards Interactive Real-World Object Retrieval

2026-07-14

AI Summary

This work addresses the limitations of existing 3D language field methods in real-world scenes, where feature compression leads to poor instance discriminability and instability under ambiguous queries. To overcome these challenges, the authors propose a scene-specific 3D language field based on Gaussian splatting, which leverages metric learning to construct a unified feature space that enhances instance-level visual discrimination. The model is trained jointly on multi-view images and multi-text labels containing ambiguous descriptions, explicitly modeling semantic ambiguity. This approach uniquely integrates instance discriminability with ambiguity awareness into a 3D language field, enabling active detection of vague user queries and interactive clarification. Evaluated under open-vocabulary settings in real scenes, the method significantly improves object retrieval accuracy while robustly handling semantic ambiguity.

Abstract

We propose Scene-specific Ambiguity-aware 3D Language Fields (SaaF), a novel Gaussian Splatting-based 3D language field designed for interactive object retrieval in a given real-world scene. Interactive object retrieval using natural language is a crucial capability for service robots operating in complex real-world environments. While recent 3D language field methods for object retrieval establish associations between rendered pixels and autoencoder-compressed CLIP features, they suffer from two limitations: (1) reduced discriminability among similar objects due to feature compression, and (2) poor handling of ambiguous queries, often resulting in unstable or incorrect retrieval. To address these limitations, SaaF introduces a metric learning strategy to construct a unified feature space that is both instance-discriminative and ambiguity-aware. (i) To enhance instance-level visual discrimination, SaaF employs metric learning that pulls image features from multiple viewpoints of the same object closer together in the feature space. (ii) To establish ambiguity awareness, the model jointly trains on multiple text labels generated by the proposed method from each tracked object image sequence, including ambiguous descriptions, to learn the semantic relationships between ambiguous and specific features in a target scene. This feature space enables fine-grained visual understanding while allowing the system to estimate query ambiguity and interactively request clarification when needed. Experimental results demonstrate that SaaF not only improves retrieval accuracy over previous methods but also robustly detects and handles ambiguity in the user text queries under open-vocabulary settings.

Problem

• 3D language fields
• object retrieval
• feature compression
• ambiguous queries
• instance discrimination

Innovation

• 3D language fields
• Gaussian Splatting
• metric learning
• ambiguity-aware retrieval
• interactive object retrieval