CrisiText: A dataset of warning messages for LLM training in emergency communication

2025-10-10

AI Summary

To address the insufficient warning message generation capability in emergency communications during crisis scenarios, this paper introduces CrisiText—the first large-scale, multi-scenario natural language generation (NLG) dataset for crisis alerting, encompassing 13 crisis categories and over 400,000 high-quality warning texts. Methodologically, we propose an event-chain construction approach and an expert-guided factual consistency mechanism to enable controllable and trustworthy message generation, alongside diverse suboptimal prompts to facilitate preference learning and robust evaluation. We conduct experiments using supervised fine-tuning, preference alignment, and zero-/few-shot learning, augmented by an automated post-editing module to enhance safety and accuracy. Results demonstrate strong in-distribution and out-of-distribution generalization, significantly advancing the practical deployment of large language models in mission-critical emergency communication systems.

Technology Category

• Natural Language Processing: Generation
• Planning, Routing, and Scheduling: Planning with Language Models
• Multiagent Systems: Agent Communication

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

Effectively identifying threats and mitigating their potential damage during crisis situations, such as natural disasters or violent attacks, is paramount for safeguarding endangered individuals. To tackle these challenges, AI has been used in assisting humans in emergency situations. Still, the use of NLP techniques remains limited and mostly focuses on classification tasks. The significant potential of timely warning message generation using NLG architectures, however, has been largely overlooked. In this paper we present CrisiText, the first large-scale dataset for the generation of warning messages across 13 different types of crisis scenarios. The dataset contains more than 400,000 warning messages (spanning almost 18,000 crisis situations) aimed at assisting civilians during and after such events. To generate the dataset, we started from existing crisis descriptions and created chains of events related to the scenarios. Each event was then paired with a warning message. The generations follow experts' written guidelines to ensure correct terminology and factuality of their suggestions. Additionally, each message is accompanied by three suboptimal warning types to allow for the study of different NLG approaches. To this end, we conducted a series of experiments comparing supervised fine-tuning setups with preference alignment, zero-shot, and few-shot approaches. We further assessed model performance in out-of-distribution scenarios and evaluated the effectiveness of an automatic post-editor.

Problem

• Generating timely warning messages for crisis situations using NLG architectures
• Creating a large-scale dataset for emergency communication across 13 crisis types
• Evaluating NLG approaches for warning message generation in out-of-distribution scenarios

Innovation

• Created large-scale crisis warning message dataset
• Generated expert-guided messages for 13 crisis types
• Compared fine-tuning with zero-shot and few-shot approaches