GeoOutageBench: Benchmarking Ambiguity-aware, Ontology-grounded Geospatiotemporal KGQA for Multimodal Power Outage and Resilience Analysis

2026-09-28

AI Summary

This study addresses the lack of benchmarks for evaluating large language models (LLMs) in handling ambiguity and ontological reasoning within multimodal spatiotemporal knowledge graph question answering (KGQA) during power outage scenarios. To bridge this gap, we introduce the first multimodal spatiotemporal KGQA benchmark tailored for infrastructure resilience, integrating visual, textual, and structured data to construct spatiotemporal knowledge graphs. We further propose a hierarchical capability query taxonomy alongside a configurable evaluation framework that systematically assesses LLMs through domain ontology modeling, natural language-to-SPARQL parsing, and spatiotemporal co-occurrence analysis. By open-sourcing the benchmark dataset, source code, and evaluation tools, this work establishes foundational design principles and a rigorous assessment basis for deploying LLM-KG systems in real-world resilience analysis applications.

Abstract

We introduce GeoOutageBench, a benchmark for assessing LLM-based geospatiotemporal KGQA for multimodal outage and resilience analysis. Unlike existing KGQA benchmarks for Web knowledge, GeoOutageBench considers a spatiotemporal KG that integrates visual, textual, and structured data from outage records, remote sensing, weather observations, storm and power events, geographic entities, and domain ontologies. It provides a competency query taxonomy at different difficulty levels from spatiotemporal containment and proximity, spatiotemporal co-occurrence analysis, multimodal evidence, to hypothetical evaluation. Over multimodal KG and query classes, GeoOutageBench provides user-configurable evaluation of three important, highly coherent yet less studied tasks: (1) LLMs' understanding for ambiguous geospatiotemporal questions in terms of NL to SPARQL interpretation, (2) query-driven assessment of ontology utility, and (3) answer accuracy of multimodal KGQA retrieval. GeoOutageBench provides a design principle and foundation for assessing LLM-KG systems that support real-world infrastructure resilience analysis. Our benchmark, source code, data, results, and other documentation are available at https://github.com/UCF-SAGE/GeoOutageBench.

Problem

• Geospatiotemporal KGQA
• Multimodal Power Outage Analysis
• LLM Benchmarking
• Ontology-grounded QA
• Infrastructure Resilience

Innovation

• Geospatiotemporal KGQA
• Multimodal Knowledge Graph
• Ambiguity-aware Benchmark
• Ontology-grounded Evaluation
• Infrastructure Resilience