Wild2Avatar: Rendering Humans Behind Occlusions

2023-12-31

arXiv.org

AI Summary

Reconstructing dynamic human appearances from monocular in-the-wild videos under occlusion remains challenging, as existing methods assume unoccluded, ideal conditions and fail to generalize to realistic occlusion scenarios. To address this, we propose an occlusion-aware end-to-end neural rendering framework. Our key contributions are: (1) the first occlusion-aware scene parameterization, explicitly decomposing the scene into three disentangled components—human body, occluders, and background; (2) a multi-objective loss function jointly optimizing human geometric completeness and decomposition robustness; and (3) a unified architecture integrating implicit geometric representations, differentiable rasterization, and occlusion-aware radiance field modeling. Extensive experiments on real-world occluded video sequences demonstrate state-of-the-art performance in novel-view synthesis and free-viewpoint rendering, significantly outperforming prior approaches in both visual quality and geometric fidelity.

Technology Category

• Computer Vision: 3D Computer Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Fairness, privacy, and diversity in economic environments
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Rendering the visual appearance of moving humans from occluded monocular videos is a challenging task. Most existing research renders 3D humans under ideal conditions, requiring a clear and unobstructed scene. Those methods cannot be used to render humans in real-world scenes where obstacles may block the camera's view and lead to partial occlusions. In this work, we present Wild2Avatar, a neural rendering approach catered for occluded in-the-wild monocular videos. We propose occlusion-aware scene parameterization for decoupling the scene into three parts - occlusion, human, and background. Additionally, extensive objective functions are designed to help enforce the decoupling of the human from both the occlusion and the background and to ensure the completeness of the human model. We verify the effectiveness of our approach with experiments on in-the-wild videos.

Problem

• Rendering humans from occluded monocular videos
• Decoupling scenes into occlusion, human, background
• Ensuring human model completeness in occluded views

Innovation

• Occlusion-aware scene parameterization for decoupling
• Extensive objective functions enforce human decoupling
• Neural rendering for occluded monocular videos