GeoBenchLLM: A Comprehensive Benchmark for Evaluating LLMs on Geo-Related Tasks

2026-08-07

AI Summary

This work addresses the lack of systematic evaluation of large language models (LLMs) on geospatial and temporal reasoning tasks. To bridge this gap, we propose GeoBenchLLM—the first unified benchmark specifically designed for geographic tasks—integrating 12 publicly available geospatial datasets spanning diverse domains and task types, thereby overcoming the limitations of prior homogeneous evaluations. Through comprehensive experiments across multiple mainstream LLMs, our study reveals a significant influence of model scale and reasoning capabilities on performance in geographic understanding. The benchmark is publicly released to foster interdisciplinary research at the intersection of geographic intelligence and large language models.

Abstract

In the context of geodata, existing Large Language Models have often been studied in a homogeneous setting, which has considerably limited insights into their generalization capabilities. In this paper, we present \benchName, a comprehensive benchmark for probing LLMs on geo-related tasks. We leverage a careful selection of twelve publicly available datasets from diverse geo-related tasks and domains, and evaluate a set of LLMs on geo-spatial and temporal understanding using our benchmark. Our results show that reasoning and size have a strong impact on overall performance. GeoBenchLLM is publicly available at https://github.com/Rfr2003/GeoBenchLLM.

Problem

• Large Language Models
• geospatial understanding
• benchmarking
• generalization capability
• geo-related tasks

Innovation

• GeoBenchLLM
• large language models
• geospatial understanding
• benchmark
• temporal reasoning