HARMON-E: Hierarchical Agentic Reasoning for Multimodal Oncology Notes to Extract Structured Data

2025-12-22

AI Summary

Extracting structured clinical data from unstructured, real-world oncology electronic health record (EHR) notes—including scanned PDFs—remains a significant challenge. Method: We propose the first patient-level integrated hierarchical LLM agent framework, incorporating modular task decomposition, context-aware retrieval-augmented generation, and multi-source contradictory information reconciliation to enable end-to-end abstraction of 103 clinical variables. Results: Evaluated on 2,250 patients and over 400,000 documents, the framework achieves a mean F1 score of 0.93; 100 variables attain F1 > 0.85, and critical variables—including biomarkers and medications—achieve F1 > 0.95. Human expert validation yields a 94% pass rate, substantially reducing annotation burden. This work establishes a highly reliable, scalable paradigm for clinical information extraction, directly supporting precision oncology and real-world evidence generation.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Information Extraction
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Unstructured notes within the electronic health record (EHR) contain rich clinical information vital for cancer treatment decision making and research, yet reliably extracting structured oncology data remains challenging due to extensive variability, specialized terminology, and inconsistent document formats. Manual abstraction, although accurate, is prohibitively costly and unscalable. Existing automated approaches typically address narrow scenarios - either using synthetic datasets, restricting focus to document-level extraction, or isolating specific clinical variables (e.g., staging, biomarkers, histology) - and do not adequately handle patient-level synthesis across the large number of clinical documents containing contradictory information. In this study, we propose an agentic framework that systematically decomposes complex oncology data extraction into modular, adaptive tasks. Specifically, we use large language models (LLMs) as reasoning agents, equipped with context-sensitive retrieval and iterative synthesis capabilities, to exhaustively and comprehensively extract structured clinical variables from real-world oncology notes. Evaluated on a large-scale dataset of over 400,000 unstructured clinical notes and scanned PDF reports spanning 2,250 cancer patients, our method achieves an average F1-score of 0.93, with 100 out of 103 oncology-specific clinical variables exceeding 0.85, and critical variables (e.g., biomarkers and medications) surpassing 0.95. Moreover, integration of the agentic system into a data curation workflow resulted in 0.94 direct manual approval rate, significantly reducing annotation costs. To our knowledge, this constitutes the first exhaustive, end-to-end application of LLM-based agents for structured oncology data extraction at scale

Problem

• Extracts structured oncology data from unstructured clinical notes
• Handles patient-level synthesis across many contradictory documents
• Reduces manual annotation costs in data curation workflows

Innovation

• Hierarchical agentic framework with modular adaptive tasks
• LLM reasoning agents with context-sensitive retrieval
• Iterative synthesis for patient-level data extraction