LLM-to-Phy3D: Physically Conform Online 3D Object Generation with LLMs

2025-06-11

AI Summary

Current LLM-to-3D approaches lack physics-aware modeling capabilities, often generating objects that violate real-world physical constraints—rendering designs infeasible. To address this, we propose the first large language model (LLM)-driven online physics-compliant 3D generation framework. Our method integrates differentiable physics simulation, a multimodal vision–physics evaluator, and an iterative prompt refinement mechanism within a black-box optimization loop, enabling joint optimization of geometric novelty and physical feasibility. The core innovation is an end-to-end closed-loop feedback system that explicitly optimizes for physical performance metrics—dynamically guiding the LLM to synthesize 3D structures satisfying mechanical and stability constraints. Evaluated on vehicle design tasks, our framework achieves a 4.5%–106.7% improvement in physics compliance rate over state-of-the-art baselines. This work establishes a new paradigm for trustworthy, AI-driven engineering design.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Sampling/Simulation-based Search
• Computer Vision: Low Level & Physics-based Vision

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

The emergence of generative artificial intelligence (GenAI) and large language models (LLMs) has revolutionized the landscape of digital content creation in different modalities. However, its potential use in Physical AI for engineering design, where the production of physically viable artifacts is paramount, remains vastly underexplored. The absence of physical knowledge in existing LLM-to-3D models often results in outputs detached from real-world physical constraints. To address this gap, we introduce LLM-to-Phy3D, a physically conform online 3D object generation that enables existing LLM-to-3D models to produce physically conforming 3D objects on the fly. LLM-to-Phy3D introduces a novel online black-box refinement loop that empowers large language models (LLMs) through synergistic visual and physics-based evaluations. By delivering directional feedback in an iterative refinement process, LLM-to-Phy3D actively drives the discovery of prompts that yield 3D artifacts with enhanced physical performance and greater geometric novelty relative to reference objects, marking a substantial contribution to AI-driven generative design. Systematic evaluations of LLM-to-Phy3D, supported by ablation studies in vehicle design optimization, reveal various LLM improvements gained by 4.5% to 106.7% in producing physically conform target domain 3D designs over conventional LLM-to-3D models. The encouraging results suggest the potential general use of LLM-to-Phy3D in Physical AI for scientific and engineering applications.

Problem

• Generating physically viable 3D objects with LLMs
• Addressing lack of physical constraints in LLM-to-3D models
• Enhancing AI-driven generative design for engineering applications

Innovation

• Online black-box refinement loop for LLMs
• Synergistic visual and physics-based evaluations
• Iterative prompt discovery for physical conformity