Stereo Anything: Unifying Stereo Matching with Large-Scale Mixed Data

2024-11-21

arXiv.org

AI Summary

Existing stereo matching models exhibit poor generalization to unseen domains, primarily due to insufficient diversity in training data. Method: We propose a general-purpose stereo matching foundation model for open-world scenarios, introducing a novel hybrid-data scaling paradigm: large-scale synthetic stereo pairs are constructed by fusing real annotated stereo images with monocular-image-based generation, augmented via controllable multi-baseline, multi-view, and multi-scene synthesis. A unified end-to-end architecture is adopted, integrating self-supervised pair generation, cross-domain mixed training, and zero-shot transfer evaluation. Contribution/Results: The model achieves strong zero-shot generalization across five public benchmarks—adapting robustly to novel environments and sensor configurations without fine-tuning. Code is publicly available.

Technology Category

• Computer Vision: Computational Photography, Image & Video Synthesis
• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Stereo matching has been a pivotal component in 3D vision, aiming to find corresponding points between pairs of stereo images to recover depth information. In this work, we introduce StereoAnything, a highly practical solution for robust stereo matching. Rather than focusing on a specialized model, our goal is to develop a versatile foundational model capable of handling stereo images across diverse environments. To this end, we scale up the dataset by collecting labeled stereo images and generating synthetic stereo pairs from unlabeled monocular images. To further enrich the model's ability to generalize across different conditions, we introduce a novel synthetic dataset that complements existing data by adding variability in baselines, camera angles, and scene types. We extensively evaluate the zero-shot capabilities of our model on five public datasets, showcasing its impressive ability to generalize to new, unseen data. Code will be available at url{https://github.com/XiandaGuo/OpenStereo}.

Problem

• Enhancing zero-shot generalization in stereo matching
• Mitigating dataset bias with mixed-data training
• Unifying diverse stereo sources for robust learning

Innovation

• Unifying heterogeneous stereo data sources
• Generating synthetic stereo from monocular images
• Scalable mixed-data paradigm for generalization