CoRef-GS: Cooperative Referring Gaussian Splatting for Multi-Agent Scene Understanding

2026-09-17

AI Summary

研究解决了多机器人场景理解中语言查询定位问题，提出CoRef-GS框架，通过几何和语义一致性对齐局部高斯地图并使用视图条件关系图进行查询定位。

Abstract

Referring scene understanding for embodied robots requires grounding object- and relation-centric language queries from a designated viewpoint. While a local semantic Gaussian map can support such grounding within one agent's observations, cooperative settings require this ability to remain effective after independently reconstructed maps are aligned and fused. In this setting, the referred target or its contextual landmark may come from another agent's observations, while spatial relations must still be interpreted from the querying robot's viewpoint. We formulate this problem as cooperative referring Gaussian grounding over fused maps, which requires geometric alignability, instance-level semantic comparability, and view-conditioned relation reasoning. Existing language-aware Gaussian methods mainly focus on single-map querying, whereas Gaussian registration methods optimize geometric or photometric alignment without preserving language-grounding-oriented semantic compatibility. We propose CoRef-GS, a cooperative referring Gaussian splatting framework. CoRef-GS constructs local open-vocabulary instance-aware Gaussian maps, then aligns partially overlapping maps with a cross-agent alignment module by geometric and semantic consistency, and grounds queries using a view-conditioned mask relation graph. We further introduce CoQuad-Ref, a dual-quadruped benchmark spanning both real-world and simulated indoor scenes. Experiments show that, on simulated scenes, CoRef-GS reduces the rotation error from 2.58° after coarse initialization to 0.15° after refinement, and improves real-world referring mIoU over ReferSplat from 52.6% to 68.8%. The established benchmark and source code will be publicly released at https://github.com/ruojiruoli17/CoRef-GS.git.

Problem

• cooperative referring
• Gaussian splatting
• multi-agent scene understanding
• map alignment
• view-conditioned relation reasoning

Innovation

• cooperative referring Gaussian splatting
• multi-agent scene understanding
• cross-agent alignment
• view-conditioned mask relation graph
• open-vocabulary instance-aware Gaussian maps