ELMTEX: Fine-Tuning Large Language Models for Structured Clinical Information Extraction. A Case Study on Clinical Reports

2025-02-08

AI Summary

This study addresses interoperability challenges in European healthcare systems by proposing a large language model (LLM)-based approach for multilingual clinical text information extraction. To process unstructured clinical reports, we design an end-to-end extraction pipeline with an interactive interface and jointly apply prompt engineering and supervised fine-tuning to adapt medium- and small-scale LLMs. Our key contributions are threefold: (1) We release the first bilingual clinical summarization dataset—comprising 60,000 English and 24,000 German samples—validated via multi-dimensional automatic and human annotation; (2) We propose a multi-granularity evaluation framework integrating ROUGE, BERTScore, and entity-level metrics; (3) Our fine-tuned model achieves 89.3% F1 on critical entity extraction—outperforming zero-shot LLMs by 22.7%—while accelerating inference by 3.8× and reducing GPU memory consumption by 64%, significantly enhancing clinical deployability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Summarization
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Europe's healthcare systems require enhanced interoperability and digitalization, driving a demand for innovative solutions to process legacy clinical data. This paper presents the results of our project, which aims to leverage Large Language Models (LLMs) to extract structured information from unstructured clinical reports, focusing on patient history, diagnoses, treatments, and other predefined categories. We developed a workflow with a user interface and evaluated LLMs of varying sizes through prompting strategies and fine-tuning. Our results show that fine-tuned smaller models match or surpass larger counterparts in performance, offering efficiency for resource-limited settings. A new dataset of 60,000 annotated English clinical summaries and 24,000 German translations was validated with automated and manual checks. The evaluations used ROUGE, BERTScore, and entity-level metrics. The work highlights the approach's viability and outlines future improvements.

Problem

• Extract structured clinical information
• Fine-tune Large Language Models
• Enhance healthcare system interoperability

Innovation

• Fine-tuning Large Language Models
• Structured clinical data extraction
• Smaller models surpass larger ones