Evaluating virtual-control-augmented trials for reproducing treatment effect from original RCTs

2025-07-21

AI Summary

This study investigates the feasibility and risks of augmenting randomized controlled trial (RCT) control arms with AI-generated synthetic patient data. Using CTGAN and TVAE models, we generated synthetic patients from real control-arm data in the IST and IST3 trials and simulated data completion under varying recruitment rates to assess bias in treatment effect estimation—specifically, absolute risk difference. Our empirical analysis reveals a critical limitation: synthetic controls generated exclusively from within-trial control data induce severe estimation bias—yielding relative biases of 133% and 76% in IST and IST3, respectively. This is the first empirical demonstration that “pure within-trial synthetic controls” are fundamentally inadequate for causal inference. The findings provide a methodological boundary and a key cautionary insight for the use of synthetic patients in RCTs, addressing a critical gap in empirical validation for synthetic data applications in causal estimation.

Technology Category

• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Machine Learning: Causal Learning
• Reasoning under Uncertainty: Causality

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

This study investigates the use of virtual patient data to augment control arms in randomised controlled trials (RCTs). Using data from the IST and IST3 trials, we simulated RCTs in which the recruitment in the control arms would stop after a fraction of the initially planned sample size, and would be completed by virtual patients generated by CTGAN and TVAE, two AI algorithms trained on the recruited control patients. In IST, the absolute risk difference(ARD) on death or dependency at 14 days was -0.012 (SE 0.014). Completing the control arm by CTGAN-generated virtual patients after the recruitment of 10% and 50% of participants, yielded an ARD of 0.004 (SE 0.014) (relative difference 133%) and -0.021 (SE 0.014) (relative difference 76%), respectively. Results were comparable with IST3 or TVAE. This is the first empirical demonstration of the risk of errors and misleading conclusions associated with generating virtual controls solely from trial data.

Problem

• Investigates virtual patient data to augment RCT control arms
• Evaluates AI-generated virtual controls' impact on treatment effect accuracy
• Demonstrates risks of errors from virtual controls in trials

Innovation

• Virtual patients augment control arms in RCTs
• CTGAN and TVAE generate AI-based virtual controls
• Empirical demonstration of virtual control risks