Model quality in football: Quantifying the quality of an Expected Threat model

2026-04-22

AI Summary

This study addresses the absence of systematic quality evaluation methods for existing Expected Threat (xT) models in football analytics, which can lead to their misuse in critical applications such as player assessment. For the first time, we establish a quantifiable framework for evaluating xT model quality by integrating Markov chain theory, Monte Carlo simulations, and expert knowledge. Our analysis reveals that model errors approximately follow a log-normal distribution and quantifies how the number of game states and training sample size influence error magnitude. Building on these insights, we propose an error threshold and practical validation criteria to ensure reliable deployment in real-world scenarios—such as scouting—and demonstrate that our framework is generalizable to other football analytics models lacking ground-truth labels.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Knowledge Representation and Reasoning: Qualitative Reasoning

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Models for Web evolution

Abstract

The recent growth in data availability in football has increased the risk of incorrect use of data-driven models, making guidelines on their validation and application necessary. The Expected Threat (xT) model is an accessible option for football organizations that start building in-house methods, yet little is known about how to assess its quality. The aim of this study is twofold: to examine how the model error depends on the number of game states and the number of training points, and to translate these results into guidelines for constructing and applying the model. Using the Markov chain underlying the model, we perform theoretical analyses and simulations to study the model error. These show that the model error is approximately log-normally distributed for a specified number of training points and game states. Additionally, we combine the simulations with expert consultation to establish the model error beyond which player evaluations based on the Expected Threat model become unreliable for scouting applications. From this, we derive rules of thumb to ensure the quality of an Expected Threat model before application, and we illustrate through an example how a validated model can be applied in practice. Because the approach generalizes to Expected Possession Value models, this paper illustrates a framework to systematically quantify model quality, despite the ground truth being unobservable in football analytics.

Problem

• Expected Threat
• model quality
• football analytics
• model validation
• scouting

Innovation

• Expected Threat
• model validation
• Markov chain
• model error
• football analytics