Assessing the Impact of Covariate Distribution and Positivity Violation on Weighting-Based Indirect Comparisons: a Simulation Study

2025-07-16

AI Summary

This study systematically evaluates the impact of covariate distributional skewness (e.g., bimodality, insufficient overlap) and violations of the positivity assumption on anchored and unanchored weighted indirect comparison methods—specifically Matching-Adjusted Indirect Comparison (MAIC) and Propensity Score Weighting (PSW). Method: We conducted extensive Monte Carlo simulations and validated findings using real-world data. Contribution/Results: Unanchored MAIC-1 demonstrated robustness—remaining unbiased with narrower confidence intervals—under moderate positivity violations and non-normal covariates, significantly outperforming MAIC-2 and PSW. However, all methods exhibited severe bias under omitted-confounder scenarios. This is the first study to empirically demonstrate MAIC-1’s relative resilience across multiple realistic challenges, providing evidence supporting its cautious application in unanchored settings. The results underscore the critical importance of formal positivity assumption assessment and rigorous confounding control in indirect treatment comparisons.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Humans and AI: Planning and Decision Support for Human-Machine Teams
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Population-Adjusted Indirect Comparisons (PAICs) are used to estimate treatment effects when direct comparisons are infeasible and individual patient data (IPD) are only available for one trial. Among PAIC methods, Matching-Adjusted Indirect Comparison (MAIC) is the most widely used. However, little is known about how MAIC performs under challenging conditions such as limited covariate overlap or markedly non-normal covariate distributions. We conducted a Monte Carlo simulation study comparing three estimators: (i) MAIC matching first moment (MAIC-1), (ii) MAIC matching first and second moments (MAIC-2), and (iii) a benchmark method leveraging full IPD -- Propensity Score Weighting (PSW). We examined eight scenarios ranging from ideal conditions to situations with positivity violations and non-normal (including bimodal) covariate distributions. We assessed both anchored and unanchored estimators and examined the impact of adjustment model misspecification. We also applied these estimators to real-world data from the AKIKI and AKIKI-2 trials, comparing renal replacement therapy strategies in critically ill patients. MAIC-1 demonstrated robust performance, remaining unbiased in the presence of moderate positivity violations and non-normal covariates, while MAIC-2 and PSW appeared more sensitive to positivity violations. All methods showed substantial bias when key confounders were omitted, emphasizing the importance of correct model specification. In real-world data, a consistent trend was found with MAIC-1 showing narrower confidence intervals with positivity violation. Our findings support the cautious use of unanchored MAICs and highlight MAIC-1's resilience across moderate violations of assumptions. However, the method's limited flexibility underscores the need for careful use in real-world settings.

Problem

• Evaluates MAIC performance under covariate overlap challenges
• Assesses impact of non-normal covariate distributions on MAIC
• Examines estimator bias from model misspecification and positivity violations

Innovation

• MAIC-1 robust under non-normal covariates
• MAIC-2 sensitive to positivity violations
• Model misspecification causes substantial bias