AssemblyWorld: Rethinking 3D Assembly with General-Purpose Agents

2026-09-30

AI Summary

This study investigates whether general-purpose agents can perform precise 3D assembly solely through visual interaction without fine-tuning. To this end, we introduce AssemblyWorld, an interactive 3D environment and evaluation benchmark, along with the first assessment framework for general-purpose agent-based 3D assembly relying exclusively on visual perception. Within this framework, agents perceive geometry from 2D views and manipulate rigid components, enabling a systematic evaluation of their assembly capabilities. Our analysis reveals a significant gap between approximate structure recovery and precise reconstruction. Experimental results demonstrate that the best-performing system achieves a component accuracy of 80.9% and a complete assembly success rate of 59.4%. Furthermore, open-source models exhibit substantially inferior performance compared to their closed-source counterparts.

Abstract

The task of 3D assembly requires translating an understanding of parts and their relationships into precise spatial arrangements. Can pretrained general-purpose agents assemble objects through visual interaction without additional assembly-specific fine-tuning? To investigate this question, we introduce AssemblyWorld, an interactive 3D environment in which agents inspect rendered views and manipulate supplied rigid parts, guided by images or assembly manuals when available. Agents perceive part geometry through 2D views rather than direct access to mesh vertices or faces, while their resulting assemblies are evaluated geometrically. Building on this environment, we construct AssemblyWorldBench, comprising 100 assembly tasks across 80 objects spanning furniture, industrial assembly, and fracture reassembly. Evaluating eight agent systems reveals substantial differences in their capabilities. The strongest system achieves 80.9% part accuracy but 59.4% complete-assembly success. The evaluated open-source systems lag substantially behind their stronger closed-source peers in both execution reliability and assembly accuracy. Analyses of visual references, interaction trajectories, and failures show how agents revise assemblies while leaving residual positioning errors. AssemblyWorld provides a common setting for both assessing the capabilities of interactive assembly agents and characterizing the gap between approximate structure recovery and precise reconstruction.

Problem

• 3D assembly
• general-purpose agents
• visual interaction
• benchmark evaluation

Innovation

• 3D Assembly
• General-Purpose Agents
• Interactive Environment
• Benchmark Evaluation
• Visual Interaction