🤖 AI Summary
Current evaluations based on benchmark accuracy fail to uncover safety risks of medical large language models in real-world clinical settings. This work proposes the AI-MASLD framework, which introduces narrative stress auditing into medical LLM assessment for the first time, drawing inspiration from metabolic stress testing in hepatology. The framework employs six categories of narrative perturbation probes to conduct dual stress tests on seven models. Using multidimensional metrics—including Metabolic Index (MI), Perturbation Flip Rate (PFR), and Counterfactual Fairness Index (CFI)—the study reveals that while models perform well on clean data, their performance diverges significantly under stress. Notably, open-source models demonstrate safety profiles comparable to or better than closed-source counterparts, and medical fine-tuning paradoxically undermines logical consistency and fairness, exposing a phenomenon of pseudo-normalization.
📝 Abstract
Large language models (LLMs) are entering clinical practice based on benchmark accuracy that may fail to detect safety-relevant failure modes. Here we present AI-MASLD, a stress-audit framework that adapts the logic of metabolic stress testing from hepatology to the evaluation of clinical LLMs. Using 240 clinical cases across six narrative perturbation probes, we subjected seven models to double-stress testing and quantified performance through three indices: metabolic index (MI), perturbation flip rate (PFR), and counterfactual fairness index (CFI). Under clean baseline conditions, all models performed uniformly well. Under realistic narrative stress, performance diverged sharply, revealing two distinct stress-response phenotypes. Quantized models exhibited pseudonormalization, in which low flip rates hid functional collapse. Medical supervised fine-tuning systematically degraded logical stability, fairness, and information extraction. An open-weight model matched or exceeded proprietary alternatives on every safety dimension. These findings establish narrative stress auditing as a necessary complement to accuracy-based evaluation.