CrisisSense-LLM: Instruction Fine-Tuned Large Language Model for Multi-label Social Media Text Classification in Disaster Informatics

2024-06-16

arXiv.org

AI Summary

Current social media disaster information classification tools predominantly adopt a single-label paradigm, limiting their ability to simultaneously capture multidimensional semantics—such as event type, informational value, and humanitarian involvement—thereby constraining the efficiency and accuracy of disaster situational awareness. To address this, we propose the first instruction-tuned large language model (LLM) for disaster intelligence, specifically designed for multi-label classification. Our method systematically integrates supervised fine-tuning (SFT) with a multi-label classification loss, leveraging an open-source LLM backbone and a high-quality, human-annotated instruction dataset of disaster-related tweets. This enables fine-grained, interpretable, and multidimensional semantic understanding. Evaluated on multiple real-world disaster datasets, our model achieves a 12.7% improvement in multi-label F1 score over conventional single-label models and baseline LLMs, significantly enhancing both the accuracy of emergency information identification and the timeliness of response support.

Technology Category

• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Intelligent Robots: Multi-Robot Systems

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

In the field of crisis/disaster informatics, social media is increasingly being used for improving situational awareness to inform response and relief efforts. Efficient and accurate text classification tools have been a focal area of investigation in crisis informatics. However, current methods mostly rely on single-label text classification models, which fails to capture different insights embedded in dynamic and multifaceted disaster-related social media data. This study introduces a novel approach to disaster text classification by enhancing a pre-trained Large Language Model (LLM) through instruction fine-tuning targeted for multi-label classification of disaster-related tweets. Our methodology involves creating a comprehensive instruction dataset from disaster-related tweets, which is then used to fine-tune an open-source LLM, thereby embedding it with disaster-specific knowledge. This fine-tuned model can classify multiple aspects of disaster-related information simultaneously, such as the type of event, informativeness, and involvement of human aid, significantly improving the utility of social media data for situational awareness in disasters. The results demonstrate that this approach enhances the categorization of critical information from social media posts, thereby facilitating a more effective deployment for situational awareness during emergencies. This research paves the way for more advanced, adaptable, and robust disaster management tools, leveraging the capabilities of LLMs to improve real-time situational awareness and response strategies in disaster scenarios.

Problem

• Social Media
• Disaster Information
• Classification

Innovation

• CrisisSense-LLM
• Multi-label Classification
• Disaster Informatics