Importance of Prompt Optimisation for Error Detection in Medical Notes Using Language Models

2026-02-25

AI Summary

This study addresses the critical need for efficient automated detection of errors in medical texts, which can otherwise lead to severe diagnostic and therapeutic risks. It presents the first systematic validation of prompt optimization as a pivotal factor in medical error detection, introducing the application of Genetic-Evolutionary Pareto-based Automatic (GEPA) prompt optimization to this task. The approach is evaluated on the MEDEC benchmark using state-of-the-art large language models, including GPT-5 and Qwen3-32B. Experimental results demonstrate that GEPA optimization substantially improves model performance: GPT-5’s accuracy increases from 0.669 to 0.785, and Qwen3-32B’s from 0.578 to 0.690. These gains significantly narrow the performance gap between models and human physicians, establishing a new state-of-the-art in medical error detection.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Search and Optimization: Metareasoning and Metaheuristics
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Errors in medical text can cause delays or even result in incorrect treatment for patients. Recently, language models have shown promise in their ability to automatically detect errors in medical text, an ability that has the opportunity to significantly benefit healthcare systems. In this paper, we explore the importance of prompt optimisation for small and large language models when applied to the task of error detection. We perform rigorous experiments and analysis across frontier language models and open-source language models. We show that automatic prompt optimisation with Genetic-Pareto (GEPA) improves error detection over the baseline accuracy performance from 0.669 to 0.785 with GPT-5 and 0.578 to 0.690 with Qwen3-32B, approaching the performance of medical doctors and achieving state-of-the-art performance on the MEDEC benchmark dataset. Code available on GitHub: https://github.com/CraigMyles/clinical-note-error-detection

Problem

• medical error detection
• language models
• prompt optimisation
• clinical notes
• healthcare

Innovation

• Prompt Optimization
• Error Detection
• Medical Text
• Genetic-Pareto (GEPA)
• Language Models