AutoPartGen: Autogressive 3D Part Generation and Discovery

2025-07-17

AI Summary

This work addresses the core challenge of *how to automatically infer part types and counts from images, 2D masks, or existing 3D objects—and generate assembleable, compositional 3D reconstructions*. We propose the first end-to-end autoregressive framework for 3D part generation. Our method leverages the highly compositional latent space of 3DShape2VecSet to unify multimodal conditioning inputs (images, masks, or 3D shapes) and autoregressively predict both geometric and semantic attributes of individual parts. This enables seamless, post-processing-free part synthesis. Crucially, unlike prior approaches that decouple part discovery and generation—either during training or inference—our framework jointly performs both tasks in a single autoregressive sequence. As a result, it achieves significantly improved global shape fidelity and part-level semantic plausibility. On compositional 3D generation benchmarks, our method establishes new state-of-the-art performance.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

We introduce AutoPartGen, a model that generates objects composed of 3D parts in an autoregressive manner. This model can take as input an image of an object, 2D masks of the object's parts, or an existing 3D object, and generate a corresponding compositional 3D reconstruction. Our approach builds upon 3DShape2VecSet, a recent latent 3D representation with powerful geometric expressiveness. We observe that this latent space exhibits strong compositional properties, making it particularly well-suited for part-based generation tasks. Specifically, AutoPartGen generates object parts autoregressively, predicting one part at a time while conditioning on previously generated parts and additional inputs, such as 2D images, masks, or 3D objects. This process continues until the model decides that all parts have been generated, thus determining automatically the type and number of parts. The resulting parts can be seamlessly assembled into coherent objects or scenes without requiring additional optimization. We evaluate both the overall 3D generation capabilities and the part-level generation quality of AutoPartGen, demonstrating that it achieves state-of-the-art performance in 3D part generation.

Problem

• Generates 3D objects autoregressively from parts
• Reconstructs 3D objects from 2D images or masks
• Automatically determines part types and counts

Innovation

• Autoregressive 3D part generation model
• Leverages compositional 3D latent space
• Inputs include images, masks, or 3D objects