Knowing the Self, Understanding the World: A Dual-Cognition Benchmark for UAV Spatio-temporal Reasoning with MLLMs

2026-07-17

AI Summary

This work addresses the lack of evaluation benchmarks for multimodal large language models (MLLMs) in unmanned aerial vehicles (UAVs) that assess joint reasoning about the agent’s own state and the external environment. We propose UAV-DualCog, the first UAV spatiotemporal reasoning benchmark designed from a “dual-cognition” perspective, which jointly evaluates MLLMs on image and video tasks requiring precise spatial or temporal localization within multi-view spatiotemporal scenarios. To support this benchmark, we automatically construct a scalable dataset from semantic point clouds, encompassing diverse scenes, multiple landmarks, and over a thousand question-answer pairs, and introduce a lightweight optimization probe for structured supervised training. Experiments reveal significant bottlenecks in current MLLMs regarding self-state reasoning, viewpoint transformation, spatial localization, and temporal interval recognition, while human baselines achieve strong performance, confirming the benchmark’s validity and challenge.

Abstract

Multimodal large language models have achieved strong performance across diverse vision-language tasks, yet their capabilities in UAV scenarios remain insufficiently explored. Recent UAV-oriented benchmarks have begun to evaluate MLLMs in aerial scenarios, but they typically focus on scene understanding, event recognition, or navigation completion, rather than jointly assessing the dual-cognition capability required for UAV agents: reasoning about both the UAV's own state and the external environment in multiview spatio-temporal contexts. To address this gap, we present UAV-DualCog, a benchmark for aerial multiview spatio-temporal reasoning built on this dual-cognition perspective. UAV-DualCog includes both image and video tasks to jointly evaluate self-state and environment-state reasoning, while requiring spatial or temporal grounding beyond discrete answer prediction. We also develop an automated pipeline that constructs data from scene-level semantic point clouds, yielding a scalable benchmark with diverse scenes, hundreds of landmarks, and thousands of QA samples. Extensive evaluations show that current MLLMs remain far from reliable in UAV dual cognition. Self-state reasoning, viewpoint transformation, precise spatial grounding, and temporal interval localization are persistent bottlenecks, and additional validation with thinking/frontier models and a human baseline confirms that the benchmark is understandable to humans but challenging for existing models. We further construct UAV-DualCog-Train from disjoint scenes and show through a lightweight optimization probe that it provides useful structured supervision, suggesting its value not only as an evaluation benchmark but also as a data resource for advancing MLLM-based UAV agents. Project website and supplementary materials: https://uav-dualcog.lozumi.com

Problem

• UAV
• dual-cognition
• spatio-temporal reasoning
• multimodal large language models
• self-state reasoning

Innovation

• dual-cognition
• UAV spatio-temporal reasoning
• multimodal large language models
• spatial-temporal grounding
• automated benchmark construction