AMNESIA: A Large Scale Medical Unlearning Benchmark Suite with Disease-Informed Analysis

2026-05-28

AI Summary

This work addresses the challenge of selective forgetting in medical large language models without full retraining, a critical yet underexplored problem due to the absence of clinical evaluation benchmarks. The authors propose AMNESIA, the first large-scale open-source medical forgetting benchmark, constructed from real-world electronic health records and encompassing 11 disease categories, 8,820 patient records, and 70,560 question-answer pairs. AMNESIA supports both patient-level and disease-level forgetting evaluations and introduces a novel metric for detecting leakage of medical terminology. By distinguishing between factual recall and clinical reasoning tasks, the benchmark integrates four state-of-the-art unlearning methods. Experimental results reveal that current approaches, when erasing data from specific patients, inadvertently degrade the retention of shared knowledge within the same disease category, underscoring the urgent need for fine-grained knowledge disentanglement mechanisms.

Abstract

Medical knowledge is continuously evolving. This creates a need to update or selectively forget information encoded in already-trained medical LLMs. Machine unlearning aims to remove the influence of specific training data from a model without full retraining. Yet, existing unlearning benchmarks rely on synthetic or small-scale general data, leaving clinical unlearning understudied. We introduce AMNESIA, the first large-scale, open source benchmark for medical unlearning, with 70,560 question-answer pairs from 8,820 patient notes across 11 disease categories. AMNESIA includes both factual questions testing direct recall and reasoning questions testing clinical inference. We use it to evaluate four widely used unlearning methods at both random patient and disease-level, and introduce a new metric for detecting leakage of medical terminology. We show that unlearning individual patients erodes knowledge of others with the same condition, calling for methods that can better separate patients from shared clinical knowledge.

Problem

• medical unlearning
• large-scale benchmark
• disease-informed analysis
• machine forgetting
• clinical knowledge

Innovation

• medical unlearning
• large-scale benchmark
• disease-informed analysis
• knowledge leakage detection
• clinical reasoning