Prompt-to-Parts: Generative AI for Physical Assembly and Scalable Instructions

2025-12-10

AI Summary

This work addresses the challenge of translating natural-language design intent into physically realizable assembly instructions. We propose a novel “Brick-Bag” physical API paradigm, using LDraw as an intermediate representation to establish an element-level, assembly-oriented universal language. Our method integrates a tool-augmented large language model (LLM), LDraw-based textual modeling, Python-based procedural generation, and structured connection-constraint reasoning—ensuring geometric validity, mechanical feasibility, and sequential constructibility. Compared with pixel-based diffusion models and conventional CAD approaches, our framework significantly improves component interoperability and capability in generating complex, multi-step assembly instructions. We validate the approach across satellite, aerospace, and architectural domains, prototyping over 3,000 components; all four original designs were successfully realized as physical assemblies. The framework delivers high-fidelity, modular, and scalable assembly outputs.

Technology Category

Machine Learning: Large Multimodal Models (LMMs)Natural Language Processing: Code Generation / Program Synthesis from Natural LanguagePlanning, Routing, and Scheduling: Planning with Language Models

Application Category

Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactionsSearch and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved informationUser Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

We present a framework for generating physically realizable assembly instructions from natural language descriptions. Unlike unconstrained text-to-3D approaches, our method operates within a discrete parts vocabulary, enforcing geometric validity, connection constraints, and buildability ordering. Using LDraw as a text-rich intermediate representation, we demonstrate that large language models can be guided with tools to produce valid step-by-step construction sequences and assembly instructions for brick-based prototypes of more than 3000 assembly parts. We introduce a Python library for programmatic model generation and evaluate buildable outputs on complex satellites, aircraft, and architectural domains. The approach aims for demonstrable scalability, modularity, and fidelity that bridges the gap between semantic design intent and manufacturable output. Physical prototyping follows from natural language specifications. The work proposes a novel elemental lingua franca as a key missing piece from the previous pixel-based diffusion methods or computer-aided design (CAD) models that fail to support complex assembly instructions or component exchange. Across four original designs, this novel "bag of bricks" method thus functions as a physical API: a constrained vocabulary connecting precisely oriented brick locations to a "bag of words" through which arbitrary functional requirements compile into material reality. Given such a consistent and repeatable AI representation opens new design options while guiding natural language implementations in manufacturing and engineering prototyping.

Problem

• Generates assembly instructions from natural language descriptions
• Ensures geometric validity and buildability with discrete parts
• Bridges semantic design intent to manufacturable physical prototypes

Innovation

• Uses constrained brick vocabulary for geometric validity
• Employs LDraw representation for step-by-step instructions
• Creates physical API bridging language to manufacturable output