DVP-MVS: Synergize Depth-Edge and Visibility Prior for Multi-View Stereo

2024-12-16

arXiv.org

AI Summary

In multi-view stereo (MVS), patch deformation in textureless regions suffers from instability due to edge misalignment and visibility occlusion. To address this, we propose a robust deformation method integrating depth-edge alignment and cross-view visibility priors. Our key contributions are: (1) a novel depth-edge alignment mechanism that combines Roberts edge detection with morphological boundary refinement to enhance geometric edge consistency; (2) a visibility-aware cross-view prior, achieved by aggregating surface normals over the visible hemisphere and enforcing epipolar-constrained depth differences to optimize geometric consistency; and (3) an iterative refinement strategy guided by multi-view reprojection and visibility map modeling. Evaluated on ETH3D and Tanks & Temples benchmarks, our method achieves state-of-the-art performance, significantly improving reconstruction robustness and generalization for textureless regions.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Patch deformation-based methods have recently exhibited substantial effectiveness in multi-view stereo, due to the incorporation of deformable and expandable perception to reconstruct textureless areas. However, such approaches typically focus on exploring correlative reliable pixels to alleviate match ambiguity during patch deformation, but ignore the deformation instability caused by mistaken edge-skipping and visibility occlusion, leading to potential estimation deviation. To remedy the above issues, we propose DVP-MVS, which innovatively synergizes depth-edge aligned and cross-view prior for robust and visibility-aware patch deformation. Specifically, to avoid unexpected edge-skipping, we first utilize Depth Anything V2 followed by the Roberts operator to initialize coarse depth and edge maps respectively, both of which are further aligned through an erosion-dilation strategy to generate fine-grained homogeneous boundaries for guiding patch deformation. In addition, we reform view selection weights as visibility maps and restore visible areas by cross-view depth reprojection, then regard them as cross-view prior to facilitate visibility-aware patch deformation. Finally, we improve propagation and refinement with multi-view geometry consistency by introducing aggregated visible hemispherical normals based on view selection and local projection depth differences based on epipolar lines, respectively. Extensive evaluations on ETH3D and Tanks&Temples benchmarks demonstrate that our method can achieve state-of-the-art performance with excellent robustness and generalization.

Problem

• Addresses deformation instability in multi-view stereo reconstruction.
• Integrates depth-edge alignment and visibility prior for robust patch deformation.
• Improves accuracy in textureless areas using visibility-aware deformation techniques.

Innovation

• Depth-edge alignment using Depth Anything V2 and Roberts operator
• Visibility-aware patch deformation with cross-view depth reprojection
• Multi-view geometry consistency with hemispherical normals and epipolar lines