VectorLLM: Human-like Extraction of Structured Building Contours vis Multimodal LLMs

2025-07-07

AI Summary

To address the low efficiency, complex pipelines, and poor generalizability in vectorizing building footprints from remote sensing imagery, this paper proposes the first end-to-end multimodal large language model (MLLM) framework that directly regresses ordered building corner sequences, enabling human-like structured vectorization. The method integrates a vision backbone, learnable positional embeddings, and a large language model, trained via three stages: pretraining, supervised fine-tuning, and preference optimization. Its key contributions are: (1) the first application of MLLMs to remote sensing building extraction; (2) zero-shot cross-region and cross-type generalization; and (3) unified modeling of diverse building geometries. Evaluated on WHU, WHU-Mix, and CrowdAI benchmarks, the framework achieves AP improvements of +5.6, +7.1, and +13.6 over state-of-the-art methods, respectively, demonstrating substantial gains in both accuracy and generalization capability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Automatically extracting vectorized building contours from remote sensing imagery is crucial for urban planning, population estimation, and disaster assessment. Current state-of-the-art methods rely on complex multi-stage pipelines involving pixel segmentation, vectorization, and polygon refinement, which limits their scalability and real-world applicability. Inspired by the remarkable reasoning capabilities of Large Language Models (LLMs), we introduce VectorLLM, the first Multi-modal Large Language Model (MLLM) designed for regular building contour extraction from remote sensing images. Unlike existing approaches, VectorLLM performs corner-point by corner-point regression of building contours directly, mimicking human annotators' labeling process. Our architecture consists of a vision foundation backbone, an MLP connector, and an LLM, enhanced with learnable position embeddings to improve spatial understanding capability. Through comprehensive exploration of training strategies including pretraining, supervised fine-tuning, and preference optimization across WHU, WHU-Mix, and CrowdAI datasets, VectorLLM significantly outperformed the previous SOTA methods by 5.6 AP, 7.1 AP, 13.6 AP, respectively in the three datasets. Remarkably, VectorLLM exhibits strong zero-shot performance on unseen objects including aircraft, water bodies, and oil tanks, highlighting its potential for unified modeling of diverse remote sensing object contour extraction tasks. Overall, this work establishes a new paradigm for vector extraction in remote sensing, leveraging the topological reasoning capabilities of LLMs to achieve both high accuracy and exceptional generalization. All the codes and weights will be published for promoting community development.

Problem

• Extracting vectorized building contours from remote sensing imagery
• Overcoming limitations of multi-stage pipelines in contour extraction
• Enhancing generalization for diverse remote sensing object extraction

Innovation

• Multimodal LLM for direct building contour extraction
• Learnable position embeddings enhance spatial understanding
• Zero-shot performance on diverse remote sensing objects