RoofNet: A Global Multimodal Dataset for Roof Material Classification

2025-05-25

AI Summary

Current natural disaster vulnerability assessments are hindered by the global absence of roof material data. To address this, we introduce RoofNet—the first large-scale, multimodal roof material classification dataset—comprising over 51,500 samples across 184 geographic regions, integrating high-resolution remote sensing imagery with expert-annotated textual descriptions and structured metadata (e.g., roof shape, area, solar panel presence, and composite materials). Methodologically, we propose a geography- and material-aware prompt tuning framework, incorporating rule-based validation and human-in-the-loop annotation verification. Leveraging vision-language model (VLM) fine-tuning and geography-informed prompt engineering, our approach achieves robust generalization across diverse climate zones and building typologies. RoofNet enables applications in insurance actuarial modeling, disaster preparedness, and infrastructure policy formulation, while establishing a new benchmark for global exposure modeling and evaluating VLMs’ regional transferability.

Technology Category

• Computer Vision: Remote Sensing / Geospatial AI
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Natural disasters are increasing in frequency and severity, causing hundreds of billions of dollars in damage annually and posing growing threats to infrastructure and human livelihoods. Accurate data on roofing materials is critical for modeling building vulnerability to natural hazards such as earthquakes, floods, wildfires, and hurricanes, yet such data remain unavailable. To address this gap, we introduce RoofNet, the largest and most geographically diverse novel multimodal dataset to date, comprising over 51,500 samples from 184 geographically diverse sites pairing high-resolution Earth Observation (EO) imagery with curated text annotations for global roof material classification. RoofNet includes geographically diverse satellite imagery labeled with 14 key roofing types -- such as asphalt shingles, clay tiles, and metal sheets -- and is designed to enhance the fidelity of global exposure datasets through vision-language modeling (VLM). We sample EO tiles from climatically and architecturally distinct regions to construct a representative dataset. A subset of 6,000 images was annotated in collaboration with domain experts to fine-tune a VLM. We used geographic- and material-aware prompt tuning to enhance class separability. The fine-tuned model was then applied to the remaining EO tiles, with predictions refined through rule-based and human-in-the-loop verification. In addition to material labels, RoofNet provides rich metadata including roof shape, footprint area, solar panel presence, and indicators of mixed roofing materials (e.g., HVAC systems). RoofNet supports scalable, AI-driven risk assessment and serves as a downstream benchmark for evaluating model generalization across regions -- offering actionable insights for insurance underwriting, disaster preparedness, and infrastructure policy planning.

Problem

• Lack of global data on roofing materials for hazard vulnerability modeling
• Need for accurate roof classification to improve disaster risk assessment
• Absence of diverse multimodal datasets for vision-language modeling in infrastructure analysis

Innovation

• Multimodal dataset with EO imagery and text
• Vision-language modeling for roof classification
• Geographic- and material-aware prompt tuning