external validity assessment

Electronic health records (EHRs) are widely used in epidemiology and AI research, yet their data quality suffers from subgroup bias, systematic errors, and insufficient applicability assessment. To address these challenges, we propose a context-aware, automated medical data quality evaluation framework—the first to adapt software engineering principles of unit testing and coverage analysis to EHR validation. Our method integrates large language models (LLMs) for test case generation, medical knowledge anchoring, and research-context-driven data fitness analysis. Based on this, we develop MDPT, a tool comprising a test generator and executor. Evaluated on All of Us, MIMIC-III, and SyntheticMass datasets, MDPT generates 55–73 tests per cohort and detects 20–43 instances of data inconsistency or anomaly. The approach significantly improves both accuracy and interpretability in assessing EHR suitability for downstream research.

This study addresses the poor external calibration and limited generalizability of prognostic models under distributional shift by proposing a dual-path strategy. For model developers, it constructs covariate and outcome distributions representative of the target population via meta-analysis to train an “average optimal” model; for end users, it selects the best-performing model based on outcome distribution similarity between cohorts. The approach quantifies distributional discrepancies using Kullback–Leibler divergence, evaluates calibration performance with the Integrated Calibration Index (ICI), and assesses clinical utility through decision curve analysis (DCA). Experiments demonstrate that greater distributional divergence correlates with worse calibration, whereas meta-analysis–based weighting significantly improves calibration (p = 0.037). Moreover, models developed in cohorts with outcome distributions similar to the target population exhibit superior calibration and higher clinical net benefit.

Current medical AI evaluation benchmarks predominantly emphasize knowledge acquisition, failing to adequately capture model reliability, safety, and clinical utility in real-world settings. To address this gap, this work proposes the first systematic evaluation framework aligned with clinical workflows, encompassing end-to-end tasks such as clinical documentation, decision support, and administrative processes. The framework integrates authentic multimodal clinical data and introduces task-specific metrics to comprehensively assess generative models, multimodal systems, and AI agents. Empirical results reveal a substantial performance gap between state-of-the-art models on real-world tasks and their scores on medical knowledge exams—scoring 0.74–0.85 in documentation, 0.61–0.76 in clinical decision-making, and 0.53–0.63 in administrative tasks—highlighting the limitations of existing evaluation paradigms and underscoring the critical role of this framework in advancing the clinical deployment of medical AI.

Current health AI evaluation benchmarks lack standardized descriptions of user queries, limiting their ability to accurately reflect model applicability in real-world clinical settings. This study systematically identifies this “validity gap” and proposes adapting clinical trial reporting standards to create structured query profiles. Leveraging large language models, we automatically annotated 18,707 health-related queries from six public benchmarks using a 16-dimensional taxonomy capturing clinical context, topic, and intent. Our analysis reveals significant structural biases: existing benchmarks severely underrepresent complex diagnostic information such as laboratory tests, imaging, and raw clinical notes; safety-critical scenarios (e.g., self-harm) constitute less than 0.7% of queries; and coverage of pediatric, geriatric, and chronic disease populations is markedly insufficient—highlighting a substantial misalignment between current evaluation frameworks and actual clinical needs.

Traditional external validation struggles to disentangle the effects of model deficiencies from population distribution shifts—often termed case-mix effects—on performance degradation. To address this, this work proposes a generative-model-based framework for patient-level similarity assessment, leveraging autoencoders to quantify how closely external individuals resemble those in the development cohort. By evaluating model performance across subgroups stratified by similarity, the approach decouples true model generalizability from dataset shift. Notably, it operates without requiring access to the original development data, thereby overcoming key limitations of conventional linear validation paradigms. Experiments on both synthetic and real-world Dutch cardiac registry data demonstrate that the framework uncovers clinically relevant performance disparities masked by standard validation practices, substantially improving the accuracy of model applicability assessments.

This study addresses the challenge of ensuring rigor in causal inference under multi-source heterogeneous data fusion by proposing a structured design paradigm grounded in the target trial framework. The approach explicitly incorporates the target population and its sampling model into the causal analysis, systematically integrating external controls, generalizability, and transportability assessments through data element alignment, transparent assumption articulation, and emulation of the target trial. Its key innovation lies in anchoring the entire framework to a precise definition of the target population, thereby identifying and mitigating irreconcilable conflicts across data sources. This strategy enhances both the reliability and interpretability of causal conclusions derived from complex, real-world data ecosystems.

Current benchmarks for medical large language models struggle to accurately predict real-world deployment performance, primarily because they fail to explicitly model the implicit assumptions underlying user–model interactions. This work proposes a conceptual distinction between “task assumptions,” which can be validated from dialogue data, and “outcome assumptions,” which require integration of outcome data and behavioral studies. To systematically evaluate how these assumptions contribute to the benchmark–deployment performance gap, the authors introduce BenchmarkCards—a novel framework coupled with a phased evaluation protocol. Retrospective clinical trial analyses and behavioral studies in case applications demonstrate that both types of assumptions contribute comparably to performance discrepancies, thereby validating the framework’s operational feasibility and practical utility, and significantly enhancing benchmark validity.

Bridging the gap between general biomedical knowledge and actionable, testable hypotheses for specific experimental or clinical contexts remains a critical challenge. This work proposes SCENE, a novel framework that formalizes knowledge contextualization as an iterative search process through a dual-layer multi-agent architecture to deeply integrate knowledge-driven and data-driven reasoning. The upper-layer agent generates search directions and anchors relevant data patterns, while the lower-layer agent leverages knowledge graph guidance and multi-objective optimization to produce verifiable propositions that balance evidential strength with empirical support. Evaluated in real-world settings, SCENE successfully identified patient subgroups with heterogeneous treatment effects in clinical trials and discovered perturbation contexts with high target-response alignment in the LINCS L1000 study, significantly outperforming existing baselines. The generated hypotheses exhibit strong traceability, reproducibility, and expert verifiability.

This study addresses the challenge of estimation bias arising from multivariate measurement error in routinely collected data, such as electronic health records. The authors propose a novel approach that integrates error-contaminated full-cohort data with a validation subsample, embedding generalized raking calibration weights within the cumulative probability model (CPM) framework. This is the first method to enable efficient and robust modeling of continuous, ordinal, or mixed-type outcomes under CPM while accounting for measurement error. By combining semiparametric rank-based regression with validation subsampling, the proposed technique substantially improves estimation accuracy, as demonstrated in an application to gestational weight gain research. Empirical results show clear advantages over existing methods, confirming its effectiveness and practical utility in real-world biomedical settings.

This study addresses the lack of standardized evaluation protocols in existing methods for synthesizing health tabular data. To this end, it systematically assesses the performance of seven prominent generative models across four health datasets of varying scales, employing consistent hyperparameter tuning and joint distribution fidelity metrics to ensure a fair comparison. The work introduces a novel, unified evaluation framework that integrates multidimensional quantitative metrics with visual analytics, complemented by domain-informed medical interpretation. Through this approach, the study uncovers critical limitations of current models in adhering to clinical constraints and provides a reproducible, interpretable foundation for selecting appropriate synthetic data generators in healthcare applications.