USAI-Quant: A Quantitative Reasoning Benchmark for Vision-Language Models in Built Environments

2026-09-26

AI Summary

This study addresses the lack of quantitative reasoning evaluation for vision-language models (VLMs) in remote sensing by constructing the first AI benchmark for urban spatial quantitative analysis. The proposed benchmark integrates high-resolution remote sensing imagery with built environment metrics across 335 major U.S. cities, featuring multi-level visual question answering (VQA) tasks designed to quantitatively assess numerical reasoning capabilities. Our experiments reveal significant performance deficiencies in current state-of-the-art models on these tasks, and we provide an in-depth analysis of their underlying causes. By bridging the gap left by existing VQA benchmarks that predominantly focus on qualitative assessment, this work establishes a new paradigm for the quantitative evaluation of intelligent remote sensing interpretation.

Abstract

Large vision-language models (VLMs) have emerged as a powerful paradigm for urban and spatial AI. However, current state-of-the-art large VLMs still struggle with quantitative reasoning on remote sensing imagery. Existing benchmarks and algorithms are predominantly based on qualitative Visual Question Answering (VQA), providing limited insights into the quantitative reasoning capabilities of VLMs for built environment metrics. To address this gap, we develop Quantitative Urban and Spatial AI benchmark (USAI-Quant), the first benchmark designed to quantitatively evaluate VLM's reasoning capabilities on built environment metrics via remote sensing imagery. USAI-Quant is curated from the 335 largest U.S. cities, aligning high-resolution remote sensing images with quantitative built environment metrics. We then evaluate both general-purpose and remote sensing VLMs (RS-VLMs) by applying VQAs to tens of built environment metrics across three complexity levels. Our results reveal that current state-of-the-art models consistently fall short on numeric reasoning tasks. We further conduct in-depth analyses across models, question types, and geographic locations, uncovering insights into performance variability and task-specific challenges.

Problem

• Vision-Language Models
• Quantitative Reasoning
• Remote Sensing Imagery
• Built Environment
• Benchmark

Innovation

• Vision-Language Models
• Quantitative Reasoning
• Remote Sensing
• Benchmark
• Built Environment