Guide Me Out: A Framework to Benchmark VLM Operators Communication in Crisis Scenarios

2026-06-08

AI Summary

This study addresses a critical gap in crisis communication research, which has predominantly focused on static text classification while overlooking the spatial guidance capabilities of AI agents in dynamic, embodied scenarios. The work proposes the first evaluation framework for vision-language models (VLMs) tailored to embodied crisis response, featuring a controlled simulated evacuation environment with multiple manipulated variables. It systematically investigates the impact of narrowcasting versus broadcasting strategies, visual versus graph-structured environmental representations, and static versus moving threats on guidance efficacy. Experimental results demonstrate that narrowcasting significantly reduces civilian failure rates, visual modalities outperform graph-based representations, and moving threats consistently increase failure rates—highlighting the crucial challenge of dynamic adaptation. This framework establishes a quantifiable benchmark for evaluating embodied AI in crisis communication.

Abstract

Effective crisis response requires spatially grounded communication that bridges linguistic guidance of civilians with the physical environment, accounting for structural bottlenecks, evolving threats, and agent-specific contexts. Yet, current NLP research in crisis communication remains mainly limited to static, text-only classification settings, overlooking the critical communicative role of AI operators in dynamic, embodied scenarios. We address this gap with a novel benchmarking framework for evaluating Vision-Language Models (VLMs) tasked with guiding civilian agents through simulated evacuations. We test two communication strategies (narrowcast vs. broadcast), two environment representations (visual vs. graph-based), and two threat behaviors (static vs. moving) across nine maps of varying structural complexity. Our results show that Narrowcast consistently reduces civilian Fail rates compared to Broadcast across all difficulty levels. Guidance quality depends heavily on how the VLM operator represents the world: the visual modality drives performance, while adding an adjacency graph is model-dependent and often harmful. Moving threats raise Fail rates across all conditions as communication must continuously adapt over time. Together, these findings show that deploying VLMs as AI operators in evacuation scenarios remains a non-trivial challenge, where the choice of communication strategy and input representation can directly determine the success or failure of the intervention.

Problem

• crisis communication
• Vision-Language Models
• evacuation guidance
• spatially grounded communication
• embodied AI

Innovation

• Vision-Language Models
• crisis communication
• evacuation guidance
• narrowcast vs broadcast
• embodied AI