🤖 AI Summary
This work addresses the systematic spatial reasoning errors exhibited by large language models when generating 3D structures from natural language instructions, which often manifest as coordinate inaccuracies that undermine structural reliability. To mitigate this, the authors propose a neuro-symbolic 2.5-D decomposition approach that disentangles deterministic physical constraints—such as gravity—from the language model’s output. The model is restricted to planning layouts in a 2D plane, while a symbolic executor determines vertical stacking based on column occupancy. This strategy significantly improves construction accuracy, achieving a 94.6% average structural correctness on the Build What I Mean benchmark—surpassing GPT-4o (90.3%) and the previous state-of-the-art system (76.3%). Notably, it retains 94.5% performance on Jetson Thor AGX edge hardware. Ablation studies attribute a 50.7-percentage-point accuracy gain to the proposed method, highlighting its potential for generalization to other physically constrained assembly tasks.
📝 Abstract
Autonomous systems that build structures from natural-language instructions need reliable spatial reasoning, yet large language models (LLMs) make systematic coordinate errors when generating three-dimensional block placements. We present a neuro-symbolic pipeline based on \emph{2.5-D decomposition}: the LLM plans in the two-dimensional horizontal plane while a deterministic executor computes all vertical placement from column occupancy, eliminating an entire class of errors. On the Build What I Mean benchmark (160 rounds), GPT-4o-mini with this pipeline achieves 94.6\% mean structural accuracy across 12 independent runs, within 3.0 percentage points of the 97.6\% ceiling imposed by architect-agent errors that no builder-side improvement can address. This outperforms both GPT-4o at 90.3\% and the best competing system at 76.3\%. A controlled ablation confirms that 2.5-D decomposition is the dominant contributor, accounting for 50.7 percentage points of accuracy. The pipeline transfers directly to edge hardware: Nemotron-3 120B running locally on an NVIDIA Jetson Thor AGX matches the cloud result at 94.5\% with no prompt modifications. The underlying principle, removing deterministic dimensions from the LLM's output space, applies to any autonomous construction or assembly task where gravity or other physical constraints fix one or more degrees of freedom. A transfer experiment on 500 IGLU collaborative building tasks confirm the effect generalizes beyond the primary benchmark.