Estimate Level Adjustment For Inference With Proxies Under Random Distribution Shifts

2026-05-07

AI Summary

This work addresses the challenge of biased estimation and miscalibrated uncertainty in proxy-based statistical inference under stochastic distribution shifts, where conventional identification assumptions are often unverifiable. The authors propose a novel calibration framework operating at the estimation level that models the discrepancy between proxy and primary outcomes as a parameter-level random effect. Leveraging aggregated data from historical domains—without requiring individual-level responses—the method estimates the distribution of this bias and seamlessly integrates into existing proxy adjustment techniques. Inspired by domain adaptation, the approach circumvents reliance on stringent identification assumptions. Empirical evaluations on public benchmarks and real-world experiments demonstrate substantial improvements in both inferential accuracy and uncertainty quantification, with notable robustness even when historical domain data are limited.

Abstract

In many scientific domains, including experimentation, researchers rely on measurements of proxy outcomes to achieve faster and more frequent reads, especially when the primary outcome of interest is challenging to measure directly. While proxies offer a more readily accessible observation for inference, the ultimate goal is to draw statistical inferences about the primary outcome parameter and proxy data are typically imperfect in some ways. To correct for these imperfections, current statistical inference methods often depend on strict identifying assumptions (such as surrogacy, covariate/label shift, or missingness assumptions). These assumptions can be difficult to validate and may be violated by various additional sources of distribution shift, potentially leading to biased parameter estimates and miscalibrated uncertainty quantification. We introduce an estimate-level framework, inspired by domain adaptation techniques, to empirically calibrate proxy-based inference. This framework models the proxy-primary metric discrepancy as a random effect at the parameter level, estimating its distribution from aggregated historical observations across past domains (e.g., experiments, time periods, or distinct segments). This method avoids the requirement for retaining individual-level response data. Additionally, this adjustment can be layered on top of existing proxy-correction methods (such as prediction-powered inference or importance weighting) to account for additional biases not addressed by those corrections. To manage uncertainty when the number of historical domains is limited, we provide both a method-of-moments estimator and a domain bootstrap procedure. We further validate this approach using publicly available datasets and real-world experiments.

Problem

• proxy inference
• distribution shift
• statistical bias
• uncertainty quantification
• surrogate outcomes

Innovation

• estimate-level adjustment
• proxy inference
• random distribution shifts
• domain adaptation
• uncertainty quantification