Generating CAD Code with Vision-Language Models for 3D Designs

2024-10-07

arXiv.org

AI Summary

Large language models (LLMs) often generate CAD scripts that deviate from design intent due to geometric distortions or compilation failures. To address this, we propose CADCodeVerify—the first iterative vision-language verification framework tailored for CAD code generation. It leverages multimodal large models (e.g., GPT-4V) to automatically formulate verification-oriented visual questions, integrates CAD rendering, point-cloud Chamfer Distance (CD) evaluation, and feedback-driven multi-turn prompt optimization, enabling end-to-end refinement from natural language to high-fidelity, executable CAD code. Our contributions are threefold: (1) the first vision-closed-loop verification paradigm for CAD code generation; (2) CADPrompt—the first benchmark comprising 200 image-text–code triplets; and (3) on GPT-4, a 7.30% reduction in point-cloud CD distance and a 5.0% improvement in compilation success rate, significantly enhancing structural integrity and dimensional compliance.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Generative AI has transformed the fields of Design and Manufacturing by providing efficient and automated methods for generating and modifying 3D objects. One approach involves using Large Language Models (LLMs) to generate Computer- Aided Design (CAD) scripting code, which can then be executed to render a 3D object; however, the resulting 3D object may not meet the specified requirements. Testing the correctness of CAD generated code is challenging due to the complexity and structure of 3D objects (e.g., shapes, surfaces, and dimensions) that are not feasible in code. In this paper, we introduce CADCodeVerify, a novel approach to iteratively verify and improve 3D objects generated from CAD code. Our approach works by producing ameliorative feedback by prompting a Vision-Language Model (VLM) to generate and answer a set of validation questions to verify the generated object and prompt the VLM to correct deviations. To evaluate CADCodeVerify, we introduce, CADPrompt, the first benchmark for CAD code generation, consisting of 200 natural language prompts paired with expert-annotated scripting code for 3D objects to benchmark progress. Our findings show that CADCodeVerify improves VLM performance by providing visual feedback, enhancing the structure of the 3D objects, and increasing the success rate of the compiled program. When applied to GPT-4, CADCodeVerify achieved a 7.30% reduction in Point Cloud distance and a 5.0% improvement in success rate compared to prior work

Problem

• Verify and improve 3D objects from CAD code.
• Generate and validate CAD code using Vision-Language Models.
• Enhance 3D object structure and program success rate.

Innovation

• Uses Vision-Language Models for CAD code verification
• Introduces CADCodeVerify for iterative 3D object improvement
• Develops CADPrompt benchmark for CAD code generation