Auditing Institutional Heterogeneity for Generative AI in Patient Education: A Large-Scale Study of 102 US Transplant Handbooks

2026-06-13

arXiv.org

AI Summary

研究使用大型语言模型评估了102份美国移植手册的一致性问题，发现同中心不同器官间一致性高于跨中心相同器官，并揭示了信息缺口及主题优先级。

Abstract

Health systems are rapidly deploying generative AI assistants that answer patient questions from institution-authored education materials, on the premise that grounding in local content yields consistent guidance. Whether it does depends on a question not previously measured at scale: do the underlying documents themselves agree? We use a structured-output large language model judge to audit 5,730,465 pairwise comparisons across 102 patient-education handbooks from 23 US solid-organ transplant centers, paired with 1,115 patient-derived questions (TransplantQA). Four findings bear directly on deployment: (1) institutional editorial voice statistically transcends organ-type boundaries, with same-center handbooks agreeing across organs more than same-organ handbooks across centers (p = 0.0056); (2) information gaps fall disproportionately on topics central to underrepresented subgroups, with reproductive health showing double jeopardy: it is the single most-silent topic (82% absent) and the highest judge-rated clinical significance when present (86% high-significance disagreements); (3) divergence themes cluster into 991 topics, with immunosuppression and pregnancy timing among the highest-stakes themes; and (4) per-pair disagreement is predictable from question framing alone (AUC = 0.77). We discuss implications for deploying patient-facing generative AI in transplant care.

Problem

• generative-AI
• patient-education
• transplant-handbooks
• institutional-heterogeneity
• information-gaps

Innovation

• structured-output large-language-model
• patient-education handbooks
• TransplantQA
• information gaps
• judge-derived themes