Morpheus: Text-Driven 3D Gaussian Splat Shape and Color Stylization

2025-03-03

AI Summary

Existing text-driven 3D stylization methods struggle to achieve plausible geometric deformation, as aggressive stylization often compromises structural consistency and cross-view stability. This paper introduces the first text-driven 3D Gaussian Splatting stylization framework enabling joint, controllable editing of geometry and appearance. Our key contributions are: (1) an RGB-D-based disentangled diffusion model that independently controls shape and texture; (2) a depth-guided cross-attention mechanism coupled with feature injection to enhance geometry-aware implicit representation; and (3) a composite frame-conditioned Warp ControlNet to ensure multi-view stylistic consistency. Extensive experiments demonstrate that our method significantly outperforms state-of-the-art approaches in PSNR, SSIM, LPIPS, and user studies. Notably, it maintains robust novel-view synthesis even under strong geometric deformations.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Exploring real-world spaces using novel-view synthesis is fun, and reimagining those worlds in a different style adds another layer of excitement. Stylized worlds can also be used for downstream tasks where there is limited training data and a need to expand a model's training distribution. Most current novel-view synthesis stylization techniques lack the ability to convincingly change geometry. This is because any geometry change requires increased style strength which is often capped for stylization stability and consistency. In this work, we propose a new autoregressive 3D Gaussian Splatting stylization method. As part of this method, we contribute a new RGBD diffusion model that allows for strength control over appearance and shape stylization. To ensure consistency across stylized frames, we use a combination of novel depth-guided cross attention, feature injection, and a Warp ControlNet conditioned on composite frames for guiding the stylization of new frames. We validate our method via extensive qualitative results, quantitative experiments, and a user study. Code will be released online.

Problem

• Enables convincing geometry changes in stylized novel-view synthesis.
• Introduces RGBD diffusion model for controlled appearance and shape stylization.
• Ensures consistency across stylized frames using depth-guided techniques.

Innovation

• Autoregressive 3D Gaussian Splatting for stylization
• RGBD diffusion model controls appearance and shape
• Depth-guided cross attention ensures frame consistency