DepthMaster: Unified Monocular Depth Estimation for Perspective and Panoramic Images

2026-06-10

AI Summary

This work addresses the limited generalizability of existing monocular depth estimation methods across diverse camera types and the scarcity of densely annotated panoramic data. The authors propose DepthMaster, a framework that decomposes panoramic images into overlapping perspective patches, thereby eliminating geometric discrepancies through a unified perspective representation. By introducing a correspondence consistency loss (CCL) and leveraging virtual projection cameras as geometric priors, the method enables seamless depth stitching without modifying the backbone network. Employing a Transformer-based architecture and a hybrid training strategy, DepthMaster achieves zero-shot state-of-the-art performance across 13 diverse datasets using only a single panoramic dataset for training—marking the first demonstration of a unified model capable of metric depth estimation for both narrow-field and 360° panoramic imagery.

Abstract

While monocular depth estimation has achieved significant progress, achieving generalized metric depth estimation for both narrow field-of-view (FoV) perspectives and $360^\circ$ panoramas remains an unsolved challenge. Existing methods are often tailored to specific camera types and struggle to produce accurate metric depth that generalizes across diverse settings. This limitation stems from two key challenges: the inherent geometric discrepancy between perspective and panoramic cameras, and the scarcity of panoramic training data with metric annotations. In this work, we introduce DepthMaster, a unified metric depth estimation framework. Rather than employing specialized networks to learn spherical distortions, we reformulate the problem by decomposing panoramic images into overlapping perspective patches. Crucially, distinct from prior projection-based methods that rely on ad-hoc architectural modifications to handle boundaries, we introduce a novel Correspondence Consistency Loss (CCL) and inject virtual projection cameras as geometric priors, allowing us to seamlessly stitch the patches while avoiding specialized operators and keeping the backbone largely compatible with standard Transformer designs. This strategy also resolves the geometric differences by unifying all inputs into a canonical perspective representation, and effectively circumvents data scarcity by directly unlocking powerful metric priors from vast perspective datasets. Trained on a mixed dataset that contains only one panorama dataset, DepthMaster achieves state-of-the-art zero-shot performance on 13 diverse datasets, outperforming not only universal methods but also leading specialist models in both perspective and panoramic domains.

Problem

• monocular depth estimation
• perspective images
• panoramic images
• metric depth
• generalization

Innovation

• unified depth estimation
• perspective-panoramic unification
• Correspondence Consistency Loss
• virtual projection cameras
• zero-shot generalization