ShapeCraft: LLM Agents for Structured, Textured and Interactive 3D Modeling

2025-10-20

AI Summary

Existing NL23D approaches predominantly generate unstructured meshes, lacking hierarchical modeling capabilities and interactive support, thus hindering integration into professional artistic workflows. To address this, we propose Graph-structured Procedural Shapes (GPS), a novel representation that parses natural language instructions into editable, textured, hierarchical 3D procedural programs. We further design an LLM-based multi-agent collaborative framework enabling semantic-driven iterative refinement and real-time user intervention. Our method synergistically integrates procedural modeling, graph neural networks, and multi-agent decision-making. Quantitative and qualitative evaluations demonstrate substantial improvements over state-of-the-art methods in geometric accuracy, semantic fidelity, and editability. Generated models natively support rigging for animation, material remapping, and topology-level editing—validating GPS’s practicality and extensibility for interactive 3D content creation.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Humans and AI: Game Design — Procedural Content Generation & Storytelling
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

3D generation from natural language offers significant potential to reduce expert manual modeling efforts and enhance accessibility to 3D assets. However, existing methods often yield unstructured meshes and exhibit poor interactivity, making them impractical for artistic workflows. To address these limitations, we represent 3D assets as shape programs and introduce ShapeCraft, a novel multi-agent framework for text-to-3D generation. At its core, we propose a Graph-based Procedural Shape (GPS) representation that decomposes complex natural language into a structured graph of sub-tasks, thereby facilitating accurate LLM comprehension and interpretation of spatial relationships and semantic shape details. Specifically, LLM agents hierarchically parse user input to initialize GPS, then iteratively refine procedural modeling and painting to produce structured, textured, and interactive 3D assets. Qualitative and quantitative experiments demonstrate ShapeCraft's superior performance in generating geometrically accurate and semantically rich 3D assets compared to existing LLM-based agents. We further show the versatility of ShapeCraft through examples of animated and user-customized editing, highlighting its potential for broader interactive applications.

Problem

• Generating structured 3D models from natural language descriptions
• Improving interactivity and texture quality in text-to-3D generation
• Enhancing geometric accuracy and semantic richness of 3D assets

Innovation

• Graph-based Procedural Shape representation for structured 3D modeling
• Multi-agent framework hierarchically parses natural language input
• Iterative refinement produces textured and interactive 3D assets