Talk2Sensors: 3D Visual Grounding in Autonomous Driving via Sensor-Adaptive Physical Cue Matching

2026-08-05

AI Summary

Current 3D visual grounding methods predominantly rely on monocular images and struggle to effectively fuse complementary cues—such as texture, geometry, and motion—from heterogeneous sensors like cameras, LiDAR, and 4D mmWave radar, thereby limiting the robustness of language-guided localization. To address this, this work proposes TSFormer, a unified framework, and introduces Talk2Sensors, the first multi-sensor 3D visual grounding dataset tailored for autonomous driving. Leveraging a language-guided coarse-to-fine physical attribute-aware fusion strategy, TSFormer incorporates language-routed attribute sampling and a sparsity-preserving modality arbitration mechanism to dynamically orchestrate multimodal cues for precise target localization. Experiments demonstrate that the proposed method achieves an 8.05 mAP improvement over the strongest baseline on Talk2Sensors and attains 53.05% Acc@0.5 on Mono3DRefer, significantly advancing cross-modal 3D grounding performance.

Abstract

As a key capability for embodied intelligence, 3D visual grounding (3DVG) has been predominantly studied in indoor scenes with RGB-D or point-cloud inputs, while existing outdoor extensions largely rely on monocular images alone. Both settings fall short of real-world outdoor perception, where heterogeneous sensors capture complementary yet distinct physical properties, such as visual texture, 3D geometry, and object kinematics, that are indispensable for flexible and robust query-adaptive grounding but remain under-exploited. To bridge this gap, we introduce Talk2Sensors, the first multi-sensor 3D visual grounding dataset built upon camera, LiDAR, and 4D radar. It contains 8,682 language instructions and 20,558 referred objects, with diverse prompts explicitly aligned with sensor-specific physical cues. Furthermore, we propose TSFormer, a unified Transformer-based framework for language-guided 3D visual grounding in autonomous driving. TSFormer adopts a coarse-to-fine property-aware fusion strategy: the Language-Routed Property Sampler first performs coarse text-conditioned feature retrieval by modulating sensor sampling weights with query-level linguistic cues, while the subsequent Sparse-Preserving Modality Arbiter module conducts fine-grained modality arbitration and text-guided refinement to determine the precise referred spatial location. This design enables dynamic routing of appearance, geometry, and motion cues according to the semantic requirements of each prompt, preventing dense modalities from overwhelming sparse but critical sensor signals. Extensive experiments demonstrate that TSFormer achieves state-of-the-art performance across multiple benchmarks: it improves over the strongest baseline by 8.05 mAP on Talk2Sensors, and transfers to the monocular Mono3DRefer benchmark with 53.05\% Acc@0.5.

Problem

• 3D visual grounding
• autonomous driving
• multi-sensor fusion
• physical cue matching
• heterogeneous sensors

Innovation

• multi-sensor fusion
• 3D visual grounding
• sensor-adaptive matching
• language-guided perception
• autonomous driving