Integrating Heterogeneous Information in Randomized Experiments: A Unified Calibration Framework

2026-03-07

AI Summary

This study addresses the challenge of effectively integrating heterogeneous information sources—such as cross-layer covariates, machine learning predictions, and external historical data—into randomized experiments to improve the precision of treatment effect estimation while preserving statistical validity. To this end, the authors propose a unified calibration framework that systematically combines internal and external information through an information proxy vector and convex optimization–based calibration weights under covariate-adaptive randomization. This framework provides the first general mechanism capable of accommodating diverse heterogeneous information sources, subsuming existing covariate adjustment methods as special cases, and theoretically guarantees “harmless efficiency”—ensuring that incorporating additional information never inflates the asymptotic variance. The large-sample validity of the resulting estimator is established, and the results are extended to settings where the number of strata and information sources grows with the sample size.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

In modern randomized experiments, large-scale data collection increasingly yields rich baseline covariates and auxiliary information from multiple sources. Such information offers opportunities for more precise treatment effect estimation, but it also raises the challenge of integrating heterogeneous information coherently without compromising validity. Covariate-adaptive randomization (CAR) is widely used to improve covariate balance at the design stage, but it typically balances only a small set of covariates used to form strata, making covariate adjustment at the analysis stage essential for more efficient estimation of treatment effects. Beyond standard covariate adjustment, it is often desirable to incorporate auxiliary information, including cross-stratum information, predictions from various machine learning models, and external data from historical trials or real-world sources. While this auxiliary information is widely available, existing covariate adjustment methods under CAR primarily exploit within-stratum covariates and do not provide a coherent mechanism for integrating it. We propose a unified calibration framework that integrates such information through an information proxy vector and calibration weights defined by a convex optimization problem. The resulting estimator recovers many recent covariate adjustment procedures as special cases while providing a systematic mechanism for both internal and external information borrowing within a single framework. We establish large-sample validity and a no-harm efficiency guarantee, showing that incorporating additional information sources cannot increase asymptotic variance, and we extend the theory to settings in which both the number of strata and the number of information sources grow with the sample size.

Problem

• heterogeneous information
• randomized experiments
• covariate adjustment
• treatment effect estimation
• information integration

Innovation

• covariate-adaptive randomization
• heterogeneous information integration
• calibration framework
• treatment effect estimation
• convex optimization