MaPa: Text-driven Photorealistic Material Painting for 3D Shapes

2024-04-26

International Conference on Computer Graphics and Interactive Techniques

AI Summary

This work addresses key limitations in text-to-3D material generation—namely, heavy reliance on large-scale 3D-text paired data, limited editability, and insufficient photorealistic rendering fidelity. We propose an end-to-end framework that operates without 3D-text paired supervision. Our core innovations are threefold: (1) adopting procedural material graphs—not conventional texture maps—as the underlying material representation; (2) designing a segment-wise controlled diffusion model integrated with differentiable rendering to jointly optimize material parameters under text guidance; and (3) enabling fine-grained semantic control via geometric segmentation, text-guided 2D diffusion priors, and material graph parameter initialization. Experiments demonstrate substantial improvements over prior methods in realism, resolution, and interactive editability. The framework supports real-time, high-fidelity material synthesis and flexible, intuitive parameter adjustments—marking a significant step toward controllable, photorealistic text-driven material generation.

Technology Category

Computer Vision: Diffusion Models for VisionNatural Language Processing: GenerationHumans and AI: Game Design — Procedural Content Generation & Storytelling

Application Category

Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphsSemantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMsEconomics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

This paper aims to generate materials for 3D meshes from text descriptions. Unlike existing methods that synthesize texture maps, we propose to generate segment-wise procedural material graphs as the appearance representation, which supports high-quality rendering and provides substantial flexibility in editing. Instead of relying on extensive paired data, i.e., 3D meshes with material graphs and corresponding text descriptions, to train a material graph generative model, we propose to leverage the pre-trained 2D diffusion model as a bridge to connect the text and material graphs. Specifically, our approach decomposes a shape into a set of segments and designs a segment-controlled diffusion model to synthesize 2D images that are aligned with mesh parts. Based on generated images, we initialize parameters of material graphs and fine-tune them through the differentiable rendering module to produce materials in accordance with the textual description. Extensive experiments demonstrate the superior performance of our framework in photorealism, resolution, and editability over existing methods. Project page: https://zju3dv.github.io/MaPa

Problem

• Generate 3D mesh materials from text descriptions
• Create segment-wise procedural material graphs for editing
• Leverage 2D diffusion models without paired 3D training data

Innovation

• Generates segment-wise procedural material graphs
• Uses pre-trained 2D diffusion model bridge
• Fine-tunes via differentiable rendering module