Same Facts, Different Diagnosis: Measuring and Mitigating Narrative Anchoring in Clinical Language Models

2026-07-29

AI Summary

This study addresses the susceptibility of clinical language models to narrative anchoring bias—where diagnostic reasoning is unduly influenced by the stylistic presentation of patient histories rather than underlying clinical facts. The authors construct a dataset of 1,000 USMLE cases, each rewritten in three distinct sociolinguistic styles while preserving identical clinical content, thereby enabling the first systematic evaluation of language style as an isolated variable. They propose NarrativeShield, a three-agent framework that structurally extracts and validates clinical facts prior to diagnosis, integrating multi-model benchmarking, fact-preservation verification, chain-of-thought prompting, and explicit debiasing instructions. This approach reduces the narrative anchoring gap to near zero (−0.004 to 0.037), substantially mitigates severe decision instability, and incurs only a minimal and acceptable reduction in overall diagnostic accuracy.

Abstract

Large language models used for clinical diagnostic reasoning are sensitive to sociolinguistic register, not just clinical content. We term this failure mode Narrative Anchoring: identical clinical facts expressed in different registers cause diagnostic outputs to diverge. Unlike prior demographic-bias work, which manipulates explicit identity tokens such as race or income, our benchmark isolates register as the sole channel of variation, with no demographic marker present in any form. We construct a dataset of 1,000 USMLE clinical vignettes, each rewritten into three sociolinguistically distinct personas under an independently audited fact-preservation guarantee, verified by a separate model that never sees the generation prompt. Across seven language models spanning three architecture families and scales, Narrative Anchoring is statistically significant under direct prompting in every model tested, with a Narrative Anchoring Gap of 0.064 to 0.151. Chain-of-thought reasoning and explicit debiasing instructions reduce the bias only partially, and their apparent gains are frequently confounded by accuracy collapse. We introduce NarrativeShield, a three-agent pipeline that structurally extracts and verifies clinical facts before diagnostic reasoning begins, reducing the Narrative Anchoring Gap to near-zero ($-0.004$ to $0.037$) and achieving the lowest rate of severely unstable decisions (DSS $<$ 0.8) of any method across all models, at a modest and mechanistically expected accuracy cost for most models. A stress test using a non-instruction-tuned base model shows that executing a debiasing intervention at all is gated by zero-shot instruction-following ability, not prompt content alone. We release our dataset, human-validated for fact preservation, as a standalone resource for studying register-based clinical bias.

Problem

• Narrative Anchoring
• clinical language models
• sociolinguistic register
• diagnostic reasoning
• clinical bias

Innovation

• Narrative Anchoring
• clinical language models
• sociolinguistic register
• bias mitigation
• NarrativeShield