Query-driven Document-level Scientific Evidence Extraction from Biomedical Studies

2025-05-09

AI Summary

Extracting scientific evidence from biomedical literature for clinical research questions remains challenging—particularly when evidence is fragmented or contradictory. Method: This paper proposes URCA, a framework integrating retrieval-augmented generation (RAG) with forest plot structural parsing, augmented by clinical question-driven document clustering and evidence-focused reasoning to enable automated, literature-level evidence extraction and conflict resolution. Contribution/Results: To support this task, we introduce CochraneForest—the first annotated dataset derived from Cochrane systematic review forest plots. Experiments demonstrate that URCA achieves a 10.3% F1-score improvement over state-of-the-art methods on CochraneForest, establishing a new benchmark for automated evidence synthesis. The framework provides a scalable, clinically grounded technical pathway for evidence integration and decision support in evidence-based medicine.

Technology Category

• Knowledge Representation and Reasoning: Automated Reasoning and Theorem Proving
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Data Mining & Knowledge Management: Representing, Reasoning, and Using Provenance, Trust

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Extracting scientific evidence from biomedical studies for clinical research questions (e.g., Does stem cell transplantation improve quality of life in patients with medically refractory Crohn's disease compared to placebo?) is a crucial step in synthesising biomedical evidence. In this paper, we focus on the task of document-level scientific evidence extraction for clinical questions with conflicting evidence. To support this task, we create a dataset called CochraneForest, leveraging forest plots from Cochrane systematic reviews. It comprises 202 annotated forest plots, associated clinical research questions, full texts of studies, and study-specific conclusions. Building on CochraneForest, we propose URCA (Uniform Retrieval Clustered Augmentation), a retrieval-augmented generation framework designed to tackle the unique challenges of evidence extraction. Our experiments show that URCA outperforms the best existing methods by up to 10.3% in F1 score on this task. However, the results also underscore the complexity of CochraneForest, establishing it as a challenging testbed for advancing automated evidence synthesis systems.

Problem

• Extracting scientific evidence for clinical research questions from biomedical studies
• Addressing document-level evidence extraction with conflicting clinical data
• Improving automated evidence synthesis using retrieval-augmented generation frameworks

Innovation

• Query-driven document-level evidence extraction
• URCA framework with retrieval-augmented generation
• CochraneForest dataset with annotated forest plots