GuideCAD: A Lightweight Multimodal Framework for 3D CAD Model Generation via Prefix Embedding

2026-06-05

AI Summary

Existing multimodal 3D CAD generation methods suffer from high computational costs and low training efficiency. This work proposes a lightweight prefix embedding mechanism that leverages a mapping network to transform image embeddings into textual prefixes, which guide a pretrained large language model to effectively fuse visual and textual information for predicting CAD modeling sequences. By introducing only a small number of trainable parameters, the approach substantially reduces resource consumption and accelerates training. Experimental results on a newly constructed text-image paired dataset demonstrate that the proposed method achieves comparable 3D CAD generation quality to baseline approaches while using approximately one-quarter of the parameters and training twice as fast.

Abstract

Multi-modal approaches used for 3D CAD generation require substantial computational resources, necessitating efficient training. To address this, we propose GuideCAD, which leverages semantically rich visual-textual representations having only a small number of trainable parameters to generate 3D CAD models. Specifically, GuideCAD uses a mapping network that converts image embeddings into prefix embeddings, enabling a pretrained large language model to integrate visual and textual information. As a result, a transformer-based decoder predicts the construction sequence using the visual-textual embeddings in order to generate the 3D CAD model. For experimental evaluation, we construct a new dataset, referred to as GuideCAD, which consists of text-image pairs. Each pair includes a text prompt that represents a 3D CAD construction sequence and its corresponding 3D CAD image. Our experimental results show that GuideCAD generates comparably high-quality 3D CAD models while using approximately four times fewer parameters and achieving twice the training efficiency compared to fine-tuning approaches. We have released the source code and dataset for our method at: https://github.com/mskimS2/GuideCAD

Problem

• 3D CAD generation
• multimodal
• computational efficiency
• training efficiency
• parameter efficiency

Innovation

• prefix embedding
• lightweight multimodal framework
• 3D CAD generation
• visual-textual representation
• pretrained language model