MuViSeg: Multi-View Segment Correspondences from Dense Geometry Priors

2026-07-20

AI Summary

This work addresses the limitations of existing methods that establish image correspondences at sparse keypoints or pixel levels, which are insufficient for instance-level high-level tasks such as topological navigation. The authors propose cross-view segment matching at the instance segmentation level, leveraging 3D foundation models (e.g., MASt3R, VGGT) to extract mask-level features and introducing three matching heads: a LightGlue-style attention mechanism, multi-scale spatial fusion, and a core multi-view joint self-attention module. The latter recovers transitive correspondences, overcoming the pairwise matching bottleneck. Experiments demonstrate significant improvements: zero-shot AUPRC increases by 4.85 and 25.9 on Replica and Virtual KITTI 2, respectively; in HM3D navigation, multi-view matching raises success rates from 50% to 70%, and the LightGlue head boosts SPL from 45.7 to 59.1—marking the first effective integration of segment-level matching into downstream tasks.

Abstract

Classical image correspondence is solved at the level of sparse keypoints or dense pixels, but the systems that consume these matches - object-level mapping, topological navigation, scene-graph maintenance - reason about whole objects. Recent work narrows this gap by matchng directly at the level of instance segments: a class-agnostic segmenter partitions each image, and per-segment descriptors are obtained by pooling features from large 3D foundation models over the masks. We build on this segment-level matching paradigm and propose three learned matching heads: a LightGlue-style attention head with DoubleSoftmax scoring on frozen MASt3R descriptors; a DPT-style multi-scale fusion module that exposes layered spatial detail from the VGGT foundation model before pooling; and - as our main contribution - a multi-view extension that performs joint self-attention over segments drawn from several views at once, recovering transitive correspondences that strictly pairwise matchers cannot reach. Under a stratified zero-shot protocol on Replica and Virtual KITTI 2 with controlled viewpoint baselines from 0 deg to 180 deg, the LightGlue-style head improves over a parameter-free Sinkhorn matcher on the same MASt3R backbone by +4.85 AUPRC on Replica and +25.9 AUPRC on Virtual KITTI 2. Dropped into the RoboHop topological navigation pipeline on the Habitat-Matterport 3D (HM3D) Instance Image Navigation benchmark without retraining, our multi-view variant raises success rate from 50% to 70%, and our LightGlue-style head raises SPL from 45.7 to 59.1.

Problem

• image correspondence
• instance segmentation
• multi-view matching
• transitive correspondences
• object-level reasoning

Innovation

• multi-view matching
• segment-level correspondence
• joint self-attention
• dense geometry priors
• zero-shot transfer