Detection and Measurement of Hailstones with Multimodal Large Language Models

2025-10-07

AI Summary

This study addresses the limited spatial coverage of ground-based hail size measurements by proposing a zero-shot, multimodal large language model (MLLM)-based method for automated hail diameter estimation from social media images. Leveraging natural in-image references—such as human hands—our approach employs both single-stage and a novel two-stage prompting strategy to perform direct regression of hail diameter without model fine-tuning. Evaluated on a dataset of 474 annotated images (hail diameters: 2–11 cm), the best-performing MLLM achieves a mean absolute error of 1.12 cm; the two-stage prompting significantly enhances cross-model stability and robustness. To our knowledge, this is the first work to demonstrate the feasibility of off-the-shelf MLLMs for fine-grained physical parameter retrieval in meteorological remote sensing. The method provides a low-cost, high spatiotemporal-coverage complementary observational tool for severe convective weather monitoring.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

This study examines the use of social media and news images to detect and measure hailstones, utilizing pre-trained multimodal large language models. The dataset for this study comprises 474 crowdsourced images of hailstones from documented hail events in Austria, which occurred between January 2022 and September 2024. These hailstones have maximum diameters ranging from 2 to 11cm. We estimate the hail diameters and compare four different models utilizing one-stage and two-stage prompting strategies. The latter utilizes additional size cues from reference objects, such as human hands, within the image. Our results show that pretrained models already have the potential to measure hailstone diameters from images with an average mean absolute error of 1.12cm for the best model. In comparison to a single-stage prompt, two-stage prompting improves the reliability of most models. Our study suggests that these off-the-shelf models, even without fine-tuning, can complement traditional hail sensors by extracting meaningful and spatially dense information from social media imagery, enabling faster and more detailed assessments of severe weather events. The automated real-time image harvesting from social media and other sources remains an open task, but it will make our approach directly applicable to future hail events.

Problem

• Detecting hailstones using social media and news images
• Measuring hailstone diameters with multimodal language models
• Automating hail assessment from crowdsourced images without fine-tuning

Innovation

• Uses multimodal large language models for hailstone detection
• Employs two-stage prompting with reference objects for measurement
• Extracts dense weather data from social media images