OVEarth-Bench: Evaluating Category Breadth and Query Diversity for Open-Vocabulary Earth Observation

2026-07-29

AI Summary

Existing open-vocabulary remote sensing benchmarks are limited in both category coverage and query diversity, hindering comprehensive model evaluation. This work proposes OVEarth-Bench, the first unified evaluation framework that simultaneously addresses semantic breadth and query variability. It expands semantic coverage through a hierarchical structure of positive and negative categories and enhances query diversity by introducing three types—lexical, referential, and reasoning-based—enabling joint zero-shot assessment of mask and bounding box localization. Experiments reveal that current methods remain limited in performance, with multimodal large language models (MLLMs) achieving the best results. Moreover, broader category coverage yields more stable model rankings, confirming the benchmark’s authenticity, diversity, and scalability.

Abstract

Open-vocabulary Earth observation (EO) aims to localize geospatial concepts specified in natural language rather than a fixed label set. Existing benchmarks, however, usually cover narrow category vocabularies or limited query forms. To fill this gap, we introduce OVEarth-Bench, which extends existing evaluation in two directions: category breadth, through broad hierarchical category coverage with positive and negative expressions, and query diversity, through vocabulary, referring, and reasoning queries. The benchmark supports mask and box localization under a unified zero-shot protocol. We evaluate a broad set of general and EO-specific methods. The evaluation reveals that: (1) the performance of current methods remains limited, while broader category coverage yields more stable model rankings; (2) MLLM-based methods achieve the strongest overall performance; and (3) EO-specific methods generally underperform general models and rarely match the strongest methods. These findings provide guidance for future open-vocabulary EO method design and highlight the importance of developing more realistic, diverse, high-quality, and large-scale benchmarks for reliable evaluation. Our data and evaluation package are released at https://earth-insights.github.io/OVEarth-bench.

Problem

• open-vocabulary Earth observation
• benchmark
• category breadth
• query diversity
• zero-shot evaluation

Innovation

• open-vocabulary Earth observation
• category breadth
• query diversity
• zero-shot evaluation
• hierarchical categories