Alligat0R: Pre-Training Through Co-Visibility Segmentation for Relative Camera Pose Regression

2025-03-10

AI Summary

This paper addresses the performance degradation of relative camera pose regression on image pairs with low overlap. We propose Co-visibility Segmentation—a novel pretraining paradigm that reformulates cross-view self-supervised learning as a pixel-level visibility classification task (co-visible, occluded, or out-of-field), thereby eliminating reliance on high-overlap image pairs. Our key contributions are: (1) the first co-visibility segmentation pretraining framework explicitly designed for pose estimation; (2) end-to-end training support for image pairs of arbitrary overlap; and (3) the release of Cub3, a large-scale synthetic dataset comprising 2.5 million image pairs with dense pixel-level visibility annotations. On benchmarks including nuScenes, our method significantly outperforms CroCo, achieving up to a 12.6% improvement in pose accuracy for image pairs with overlap below 30%. Both code and the Cub3 dataset will be made publicly available.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multi-instance/Multi-view Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Data transparency and provenance
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Pre-training techniques have greatly advanced computer vision, with CroCo's cross-view completion approach yielding impressive results in tasks like 3D reconstruction and pose regression. However, this method requires substantial overlap between training pairs, limiting its effectiveness. We introduce Alligat0R, a novel pre-training approach that reformulates cross-view learning as a co-visibility segmentation task. Our method predicts whether each pixel in one image is co-visible in the second image, occluded, or outside the field of view (FOV), enabling the use of image pairs with any degree of overlap and providing interpretable predictions. To support this, we present Cub3, a large-scale dataset with 2.5 million image pairs and dense co-visibility annotations derived from the nuScenes dataset. This dataset includes diverse scenarios with varying degrees of overlap. The experiments show that Alligat0R significantly outperforms CroCo in relative pose regression, especially in scenarios with limited overlap. Alligat0R and Cub3 will be made publicly available.

Problem

• Improves relative camera pose regression accuracy
• Enables pre-training with image pairs of any overlap
• Introduces a large dataset for co-visibility segmentation

Innovation

• Co-visibility segmentation for pre-training
• Predicts pixel visibility across image pairs
• Uses large-scale dataset with diverse overlaps