Theia: Large-Scale Multimodal Captioning and Automated Validation of the Incidents1M Dataset for Data-Free Distillation

2026-07-30

AI Summary

This study addresses the scarcity of high-quality, large-scale multimodal datasets with semantically aligned image-text pairs in disaster response, which hinders data-free knowledge distillation for vision-language models. Building upon the purely visual Incidents1M dataset, the authors leverage the Qwen3.5 series of large language models to generate image captions and introduce an innovative image-blinded LLM-as-a-Judge automated verification mechanism. This approach simulates the modality gap faced by student models during distillation, effectively identifying and suppressing annotation inconsistencies and false positives inherent in human labeling. The method produces high-fidelity descriptions for 100,000 images, achieving a semantic consistency score of 78.65 on 173,179 annotated pairs and a verification accuracy of 77.6%, substantially enhancing the semantic reliability of data-free distillation.

Abstract

The deployment of Vision-Language Models (VLMs) in critical domains like disaster management requires high-quality multimodal datasets, especially for transferring knowledge via Data-Free Knowledge Distillation (DFKD). However, existing datasets in this domain either entirely lack descriptive text, such as Incidents1M, or suffer from severe text-image semantic misalignment, such as CrisisMMD. In this work, we present a novel methodology to construct and automatically validate a large-scale multimodal dataset for disaster response. Starting from the vision-only Incidents1M, we successfully recovered 100,000 images and generated high-fidelity textual descriptions using two distinct Qwen3.5 architectures: a 4B dense model and a 35B Mixture-of-Experts (MoE) model. To ensure the generated captions provide reliable semantic anchoring for DFKD, we introduce an image-blind LLM-as-a-Judge validation pipeline leveraging Qwen3.5-9B. By intentionally obscuring the original image from the judge, this evaluator accurately simulates the modality gap of the student model during data-free distillation. Our evaluation across 173,179 label pairs demonstrates a high semantic agreement (78.65/100) between the two architectures. Furthermore, the automated evaluation reveals a conservative captioning behaviour, characterized by a high Precision (77.6%) and low Recall (46.0%). This minimizes the false positive noise, while simultaneously exposing underlying human annotation inconsistencies in the original ground truth. This work provides a scalable, LLM-validated multimodal dataset and a reproducible framework to advance cross-modal knowledge distillation.

Problem

• multimodal dataset
• semantic misalignment
• data-free knowledge distillation
• disaster response
• captioning

Innovation

• Data-Free Knowledge Distillation
• Multimodal Captioning
• LLM-as-a-Judge
• Semantic Alignment
• Incidents1M