Assessing the replicability of RCTs in RWE emulations

2024-12-12

AI Summary

This study addresses the limited reproducibility of real-world evidence (RWE) in emulating randomized controlled trial (RCT) results. We propose and systematically extend the “sceptical p-value” as a statistically rigorous alternative to the conventional two-trial rule. Specifically, we generalize it to non-inferiority trials for the first time and develop a confidence-interval-based inversion framework for computation. Empirical evaluation on the RCT DUPLICATE platform demonstrates that, while strictly controlling the Type I error rate, the method substantially improves statistical power and enhances sensitivity to true treatment effects—yet yields qualitatively consistent conclusions with the two-trial rule. Our key contributions are: (1) adaptation of the sceptical p-value to non-inferiority settings and development of robust inferential procedures; and (2) establishment of a new paradigm for RWE validation that balances statistical rigor with practical feasibility.

Technology Category

• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Machine Learning: Calibration & Uncertainty Quantification
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Background: The standard regulatory approach to assess replication success is the two-trials rule, requiring both the original and the replication study to be significant with effect estimates in the same direction. The sceptical p-value was recently presented as an alternative method for the statistical assessment of the replicability of study results. Methods: We review the statistical properties of the sceptical p-value and compare those to the two-trials rule. We extend the methodology to non-inferiority trials and describe how to invert the sceptical p-value to obtain confidence intervals. We illustrate the performance of the different methods using real-world evidence emulations of randomized, controlled trials (RCTs) conducted within the RCT DUPLICATE initiative. Results: The sceptical p-value depends not only on the two p-values, but also on sample size and effect size of the two studies. It can be calibrated to have the same Type-I error rate as the two-trials rule, but has larger power to detect an existing effect. In the application to the results from the RCT DUPLICATE initiative, the sceptical p- value leads to qualitatively similar results than the two-trials rule, but tends to show more evidence for treatment effects compared to the two-trials rule. Conclusion: The sceptical p-value represents a valid statistical measure to assess the replicability of study results and is especially useful in the context of real-world evidence emulations.

Problem

• Assessing replicability of RCTs using sceptical p-value
• Comparing sceptical p-value with two-trials rule
• Extending methodology to non-inferiority trials

Innovation

• Uses sceptical p-value for replicability assessment
• Extends methodology to non-inferiority trials
• Inverts sceptical p-value for confidence intervals