Causal Evidence Extraction and Triangulation in Crisis Reports using Large Language Models: A ReliefWeb-based Study

2026-08-05

AI Summary

This study addresses the challenge of efficiently extracting causal evidence from lengthy and heterogeneous humanitarian crisis reports. The authors propose a two-stage large language model pipeline that structurally extracts intervention–outcome relationships from ReliefWeb reports (2000–2024). Their approach innovatively integrates query-constrained causal extraction, snippet-level evidence provenance, and a context-preserving triangulation framework, enabling, for the first time, auditable and quantifiable aggregation of cross-contextual causal evidence. Evaluated on 100 expert-annotated reports, the pipeline—implemented with a closed-source LLM and a supervised fine-tuned Llama-3.1-8B—achieves weighted F1 scores of 90.73% and 94.15%, respectively. An analysis of cash assistance interventions reveals strong positive convergence for food-related outcomes (Level of Evidence = 0.865) and long-term stability.

Abstract

Humanitarian reports are long, noisy, and multi-topic, making it difficult to consolidate decision-relevant causal evidence. We present a ReliefWeb study (2000-2024) and a two-stage Large Language Model (LLM) pipeline that extracts structured intervention-outcome records with direction and strength attributes. Query-conditioned extraction restricts output to a specified intervention class, reducing retrieval-induced over-extraction, while snippet grounding links each relation to supporting text for auditability and classification. In an expert-annotated dataset of 100 reports, the best closed-source LLM achieved a weighted F1 score of 90.73% with strong cost-efficiency, while Llama-3.1-8B with supervised fine-tuning reached 94.15% weighted F1 score. We further propose context-preserving triangulation that aggregates strength-weighted evidence within disaster$\times$source cells, applies Laplace smoothing and equally weights cells to quantify cross-context convergence via a Level-of-Evidence score. Applied to cash assistance, food-related outcomes show strong positive convergence (LoE=0.865) and stable long-horizon trajectories.

Problem

• causal evidence extraction
• crisis reports
• humanitarian reports
• decision-relevant evidence
• evidence consolidation

Innovation

• causal evidence extraction
• query-conditioned LLM
• snippet grounding
• context-preserving triangulation
• Level-of-Evidence scoring