PART: Learning 3D Part Assembly and Retrieval with Transformers

2026-09-17

AI Summary

本文提出PART框架，使用Transformer解决3D部件检索与装配问题，通过将任务转化为集合预测问题，并设计了一种新型的Transformer架构来应对挑战。

Abstract

3D assembly is fundamental to modern manufacturing and digital content creation. In this paper, we present PART, a unified transformer-based framework for 3D part retrieval and assembly: given a target shape and a part library, PART automatically selects the appropriate parts and predicts their 6-DoF poses to reconstruct the target. While prior work has achieved impressive progress on assembling a pre-defined set of parts, this more practical retrieval-based setting remains largely unexplored. The task faces three key challenges: (i) a combinatorially explosive search space that grows exponentially with library size; (ii) variable-length outputs, as different targets require different numbers of parts; and (iii) continuous 6-DoF pose estimation for part assembly. To address these, we formulate retrieval and assembly as a set prediction problem and design a novel transformer-based framework that retrieves parts and regresses their poses with variable-length output. Additionally, we exploit the duality between part pose estimation and target segmentation through joint training and a novel segmentation-enhanced optimization module. Finally, We curate a large-scale dataset of 80K+ shapes, and the results show that PART generalizes to scene layouts, image targets, and real-world scans. Project Page: https://iambrc.github.io/PART-project-page/.

Problem

• 3D Part Retrieval
• Assembly
• Transformer
• 6-DoF Pose Estimation
• Combinatorial Search Space

Innovation

• Transformer
• set prediction
• variable-length output
• 6-DoF pose estimation
• joint training