CoordRefer: Coordinate-Aware 3D Visual Grounding from Multiview Images

2026-08-05

AI Summary

In multi-view 3D visual grounding, the joint optimization of coordinate system selection and 3D bounding box regression often introduces ambiguity due to coordinate relativity, degrading localization performance. This work proposes CoordRefer, a novel framework that decouples these two components for the first time: it first selects a reference coordinate system and then regresses the 3D bounding box conditioned on that reference. The two stages are jointly optimized through coordinate-aware supervised fine-tuning and a group-relative strategy guided by 3D IoU rewards. Integrated with the Qwen3-VL-2B multimodal model, the method achieves an 11% improvement in Acc@0.25 and a 7% gain in Acc@0.5 on the ScanRefer benchmark; its geometry-optimized variant even surpasses existing approaches that rely on explicit 3D inputs.

Abstract

Multiview image-based 3D visual grounding predicts a coordinate frame to define a coordinate system and then regresses a 3D bounding box for localization. However, existing methods jointly optimize coordinate frame selection and box regression, leading to coordinate-relative box ambiguity and degraded grounding performance. This ambiguity arises because the same box admits different numerical representations across coordinate frames, creating multiple optimization targets and yielding invalid compromise predictions. To tackle this challenge, we propose CoordRefer, a coordinate-aware framework that decouples coordinate frame selection from coordinate-conditioned grounding. CoordRefer first selects a reference frame to define the coordinate system and then conditions 3D box prediction on the coordinate system. We perform coordinate-aware supervised fine-tuning to establish coordinate frame selection and coordinate-conditioned box regression, followed by Group Relative Policy Optimization with 3D IoU-based rewards to align both stages with downstream grounding quality. On ScanRefer with Qwen3-VL-2B, CoordRefer achieves gains of 11% in Acc@0.25 and 7% in Acc@0.5 over the coordinate-agnostic baseline, while its geometrically refined variant surpasses methods using explicit 3D inputs.

Problem

• 3D visual grounding
• coordinate ambiguity
• multiview images
• 3D bounding box regression
• coordinate frame selection

Innovation

• coordinate-aware
• 3D visual grounding
• decoupled optimization
• reference frame selection
• 3D IoU-based reward