ArcPro: Architectural Programs for Structured 3D Abstraction of Sparse Points

2025-03-04

AI Summary

Reconstructing structured 3D building models from extremely sparse and low-quality point clouds remains challenging. Method: We propose a domain-specific language (DSL)-based procedural modeling framework—the first building-oriented procedural DSL—unifying forward program synthesis and inverse program inference to enable end-to-end mapping from multi-modal inputs (e.g., point clouds, multi-view images, natural language) to parseable, editable building programs. Our architecture jointly employs a 3D convolutional encoder and a Transformer decoder to tokenize and generate program sequences. Contribution/Results: Compared to conventional proxy-based reconstruction and learning-based abstraction methods, our approach achieves significant improvements in both geometric fidelity and structural plausibility while maintaining efficient inference. It establishes a novel paradigm for editable 3D content generation from low-fidelity perceptual data.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Humans and AI: Game Design — Procedural Content Generation & Storytelling
• Machine Learning: Deep Neural Architectures and Foundation Models

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

We introduce ArcPro, a novel learning framework built on architectural programs to recover structured 3D abstractions from highly sparse and low-quality point clouds. Specifically, we design a domain-specific language (DSL) to hierarchically represent building structures as a program, which can be efficiently converted into a mesh. We bridge feedforward and inverse procedural modeling by using a feedforward process for training data synthesis, allowing the network to make reverse predictions. We train an encoder-decoder on the points-program pairs to establish a mapping from unstructured point clouds to architectural programs, where a 3D convolutional encoder extracts point cloud features and a transformer decoder autoregressively predicts the programs in a tokenized form. Inference by our method is highly efficient and produces plausible and faithful 3D abstractions. Comprehensive experiments demonstrate that ArcPro outperforms both traditional architectural proxy reconstruction and learning-based abstraction methods. We further explore its potential to work with multi-view image and natural language inputs.

Problem

• Recover structured 3D abstractions from sparse point clouds
• Map unstructured point clouds to architectural programs
• Outperform traditional and learning-based 3D abstraction methods

Innovation

• Domain-specific language for hierarchical building representation
• Encoder-decoder mapping point clouds to architectural programs
• Feedforward and inverse procedural modeling integration