GEOBench-VLM: Benchmarking Vision-Language Models for Geospatial Tasks

2024-11-28

arXiv.org

AI Summary

To address the suboptimal performance of general-purpose vision-language models (VLMs) on geospatial tasks—such as environmental monitoring and disaster response—this work introduces GeoBench, the first VLM benchmark dedicated to remote sensing understanding. GeoBench comprises over 10,000 human-verified, multi-source remote sensing instructions spanning six core tasks: scene understanding, fine-grained classification, object counting, localization, segmentation, and cross-temporal analysis. It systematically evaluates VLM capabilities on geospatially unique challenges, including detection of tiny objects, large-scale counting, and change identification. Experimental results reveal a significant capability gap: the state-of-the-art model LLaVA-OneVision achieves only 41.7% accuracy on multiple-choice tasks—substantially below its performance in general-domain benchmarks. GeoBench is publicly released to serve as a standardized evaluation platform for advancing geospatial AI research.

Technology Category

• Computer Vision: Remote Sensing / Geospatial AI
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

While numerous recent benchmarks focus on evaluating generic Vision-Language Models (VLMs), they do not effectively address the specific challenges of geospatial applications. Generic VLM benchmarks are not designed to handle the complexities of geospatial data, an essential component for applications such as environmental monitoring, urban planning, and disaster management. Key challenges in the geospatial domain include temporal change detection, large-scale object counting, tiny object detection, and understanding relationships between entities in remote sensing imagery. To bridge this gap, we present GEOBench-VLM, a comprehensive benchmark specifically designed to evaluate VLMs on geospatial tasks, including scene understanding, object counting, localization, fine-grained categorization, segmentation, and temporal analysis. Our benchmark features over 10,000 manually verified instructions and spanning diverse visual conditions, object types, and scales. We evaluate several state-of-the-art VLMs to assess performance on geospatial-specific challenges. The results indicate that although existing VLMs demonstrate potential, they face challenges when dealing with geospatial-specific tasks, highlighting the room for further improvements. Notably, the best-performing LLaVa-OneVision achieves only 41.7% accuracy on MCQs, slightly more than GPT-4o, which is approximately double the random guess performance. Our benchmark is publicly available at https://github.com/The-AI-Alliance/GEO-Bench-VLM .

Problem

• Evaluates VLMs for geospatial-specific challenges like temporal change detection.
• Assesses VLMs on tasks including object counting, localization, and segmentation.
• Highlights performance gaps in VLMs for geospatial applications.

Innovation

• GEOBench-VLM for geospatial VLM evaluation
• 10,000+ verified geospatial instructions
• Assesses VLMs on geospatial-specific tasks