🤖 AI Summary
Estimating win probabilities in sports analytics is inherently challenging due to high observational noise and strong multicollinearity among predictors, leading to biased, high-variance binary win/loss models with severely miscalibrated confidence intervals.
Method: We construct a stochastic-walk-based football simulation environment with known ground-truth win probabilities and conduct Monte Carlo experiments using multiple machine learning regressors (e.g., logistic regression, gradient boosting) to quantify estimator performance under realistic data dependencies.
Contribution/Results: We provide the first empirical quantification showing that observation-dependent structures substantially degrade estimator bias, variance, and nominal coverage. Crucially, effective sample size decays markedly, necessitating substantial widening of conventional confidence intervals to achieve target coverage. This phenomenon is generalizable across clustered sports data, offering both theoretical grounding and empirical benchmarks for characterizing the fundamental uncertainty in win-probability modeling.
📝 Abstract
Estimating win probability is one of the classic modeling tasks of sports analytics. Many widely used win probability estimators use machine learning to fit the relationship between a binary win/loss outcome variable and certain game-state variables. To illustrate just how difficult it is to accurately fit such a model from noisy and highly correlated observational data, in this paper we conduct a simulation study. We create a simplified random walk version of football in which true win probability at each game-state is known, and we see how well a model recovers it. We find that the dependence structure of observational play-by-play data substantially inflates the bias and variance of estimators and lowers the effective sample size. Further, to achieve approximately valid marginal coverage, win probability confidence intervals need to be substantially wide. Concisely, these are high variance estimators subject to substantial uncertainty. Our findings are not unique to the particular application of estimating win probability; they are broadly applicable across sports analytics, as myriad other sports datasets are clustered into groups of observations that share the same outcome.