Painting 3D Nature in 2D: View Synthesis of Natural Scenes from a Single Semantic Mask

2023-02-14

Computer Vision and Pattern Recognition

AI Summary

This work addresses the problem of generating multi-view-consistent, photorealistic color images of natural scenes from a single semantic mask, without requiring multi-view supervision or category-specific priors. Methodologically, we introduce the “Semantic Field”—a learnable intermediate representation bridging semantic masks and neural radiance fields (NeRFs)—which implicitly maps semantic layouts to 3D geometry and appearance by distilling prior knowledge from pretrained semantic-to-image synthesis models (e.g., Seg2Image). Training is fully self-supervised using only a collection of monocular Internet images. To our knowledge, this is the first framework enabling open-domain, 3D-consistent view synthesis for complex natural scenes—including intricate terrain, vegetation, and organic structures—from a single semantic mask. Experiments demonstrate significant improvements over existing baselines in both visual fidelity and cross-view consistency, producing high-fidelity, geometrically plausible videos of dynamic natural scenes.

Abstract

We introduce a novel approach that takes a single semantic mask as input to synthesize multi-view consistent color images of natural scenes, trained with a collection of single images from the Internet. Prior works on 3D-aware image synthesis either require multi-view supervision or learning category-level prior for specific classes of objects, which are inapplicable to natural scenes. Our key idea to solve this challenge is to use a semantic field as the intermediate representation, which is easier to reconstruct from an input semantic mask and then translated to a radiance field with the assistance of off-the-shelf semantic image synthesis models. Experiments show that our method outperforms baseline methods and produces photorealistic and multi-view consistent videos of a variety of natural scenes. The project website is https://zju3dv.github.io/paintingnature/.

Problem

• Synthesize multi-view images from single semantic mask
• Avoid multi-view supervision for natural scenes
• Use semantic field as intermediate representation

Innovation

• Uses semantic field as intermediate representation
• Translates semantic field to radiance field
• Leverages off-the-shelf semantic image synthesis