GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensing

2025-01-23

AI Summary

Existing large vision-language models (LVMs) for remote sensing struggle with pixel-level fine-grained understanding, particularly in multi-scale object localization and semantic grounding. This work introduces GeoPixel—the first remote sensing LVM supporting pixel-accurate localization—capable of interactive mask generation and grounded dialogues on 4K-resolution imagery with arbitrary aspect ratios. We propose a spatial-prior-guided data construction paradigm and release GeoPixelD, the first fine-grained grounded remote sensing dialogue dataset. The model integrates ensemble token prompting, geometric prior modeling, a high-resolution adapter, and grounded conversation generation (GCG) training. On both single- and multi-object remote sensing segmentation benchmarks, GeoPixel significantly outperforms state-of-the-art large multimodal models (LMMs). Ablation studies confirm the efficacy of each component. Code and data will be publicly released.

Technology Category

• Computer Vision: Remote Sensing / Geospatial AI
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Recent advances in large multimodal models (LMMs) have recognized fine-grained grounding as an imperative factor of visual understanding and dialogue. However, the benefits of such representation in LMMs are limited to the natural image domain, and these models perform poorly for remote sensing (RS). The distinct overhead viewpoint, scale variation, and presence of small objects in high-resolution RS imagery present a unique challenge in region-level comprehension. Moreover, the development of the grounding conversation capability of LMMs within RS is hindered by the lack of granular, RS domain-specific grounded data. Addressing these limitations, we propose GeoPixel - the first end-to-end high resolution RS-LMM that supports pixel-level grounding. This capability allows fine-grained visual perception by generating interleaved masks in conversation. GeoPixel supports up to 4K HD resolution in any aspect ratio, ideal for high-precision RS image analysis. To support the grounded conversation generation (GCG) in RS imagery, we curate a visually grounded dataset GeoPixelD through a semi-automated pipeline that utilizes set-of-marks prompting and spatial priors tailored for RS data to methodically control the data generation process. GeoPixel demonstrates superior performance in pixel-level comprehension, surpassing existing LMMs in both single-target and multi-target segmentation tasks. Our methodological ablation studies validate the effectiveness of each component in the overall architecture. Our code and data will be publicly released.

Problem

• Large-scale Image Understanding Models
• Remote Sensing Technology
• Object Recognition

Innovation

• GeoPixel
• Remote Sensing Imagery
• Object Localization