Lunguage: A Benchmark for Structured and Sequential Chest X-ray Interpretation

2025-05-27

AI Summary

Existing radiology report evaluation methods are limited to single-report scenarios and rely on coarse-grained metrics (e.g., BLEU, ROUGE), failing to capture fine-grained clinical semantics and temporal dependencies inherent in disease progression. Method: We introduce the first structured benchmark for both single-report and longitudinal chest X-ray report assessment, comprising 1,473 expert-annotated reports and 80 temporally annotated disease progression sequences. We propose (i) a radiology-specific structured generation benchmark, (ii) a two-stage temporal parsing framework, and (iii) LUNGUAGESCORE—a novel, interpretable metric enabling three-level semantic alignment (entity–relation–attribute) and explicit temporal consistency modeling. Contribution/Results: Empirical evaluation demonstrates that LUNGUAGESCORE significantly outperforms conventional metrics, markedly improving clinical relevance and temporal sensitivity. The benchmark and code are publicly released to foster reproducible research.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Machine Learning: Evaluation and Analysis

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Radiology reports convey detailed clinical observations and capture diagnostic reasoning that evolves over time. However, existing evaluation methods are limited to single-report settings and rely on coarse metrics that fail to capture fine-grained clinical semantics and temporal dependencies. We introduce LUNGUAGE,a benchmark dataset for structured radiology report generation that supports both single-report evaluation and longitudinal patient-level assessment across multiple studies. It contains 1,473 annotated chest X-ray reports, each reviewed by experts, and 80 of them contain longitudinal annotations to capture disease progression and inter-study intervals, also reviewed by experts. Using this benchmark, we develop a two-stage framework that transforms generated reports into fine-grained, schema-aligned structured representations, enabling longitudinal interpretation. We also propose LUNGUAGESCORE, an interpretable metric that compares structured outputs at the entity, relation, and attribute level while modeling temporal consistency across patient timelines. These contributions establish the first benchmark dataset, structuring framework, and evaluation metric for sequential radiology reporting, with empirical results demonstrating that LUNGUAGESCORE effectively supports structured report evaluation. The code is available at: https://github.com/SuperSupermoon/Lunguage

Problem

• Evaluates structured radiology report generation longitudinally
• Captures fine-grained clinical semantics and temporal dependencies
• Proposes interpretable metric for entity and temporal consistency

Innovation

• Two-stage framework for structured report generation
• Longitudinal annotations for disease progression tracking
• Interpretable metric for entity and temporal consistency