🤖 AI Summary
To address two key challenges in tech companies’ A/B testing guided by North Star Metrics—low metric sensitivity and the difficulty of inferring long-term business impact from short-term experimental results—this paper formalizes proxy metric design as a multi-objective optimization problem and proposes a Pareto-optimal proxy construction method. The method jointly optimizes predictive accuracy of long-term causal effects and statistical power of short-term hypothesis testing, integrating causal inference, experimental statistical modeling, and multi-objective optimization algorithms. Evaluated on a large-scale industrial recommendation system, the proposed proxy metrics achieve 8× higher detection sensitivity than the North Star Metric, with significantly improved directional consistency, thereby accelerating high-quality feature deployment decisions. The core contributions are: (i) a formalized trade-off framework that explicitly balances causal fidelity and statistical sensitivity, and (ii) a computationally tractable mechanism for identifying the Pareto frontier of proxy metrics.
📝 Abstract
North star metrics and online experimentation play a central role in how technology companies improve their products. In many practical settings, however, evaluating experiments based on the north star metric directly can be difficult. The two most significant issues are 1) low sensitivity of the north star metric and 2) differences between the short-term and long-term impact on the north star metric. A common solution is to rely on proxy metrics rather than the north star in experiment evaluation and launch decisions. Existing literature on proxy metrics concentrates mainly on the estimation of the long-term impact from short-term experimental data. In this paper, instead, we focus on the trade-off between the estimation of the long-term impact and the sensitivity in the short term. In particular, we propose the Pareto optimal proxy metrics method, which simultaneously optimizes prediction accuracy and sensitivity. In addition, we give an efficient multi-objective optimization algorithm that outperforms standard methods. We applied our methodology to experiments from a large industrial recommendation system, and found proxy metrics that are eight times more sensitive than the north star and consistently moved in the same direction, increasing the velocity and the quality of the decisions to launch new features.
