Evaluating financial tail risk forecasts: Testing Equal Predictive Ability

2025-05-29

AI Summary

This paper addresses the comparative evaluation of financial tail-risk forecasting models—specifically Value-at-Risk (VaR) and Expected Shortfall (ES)—at extreme quantiles (e.g., 1%). It identifies critical failures of the Diebold–Mariano (DM) test and Hansen’s Model Confidence Set (MCS) under small samples (≤2 years) and highly asymmetric loss functions. Integrating Gneiting’s framework of strictly consistent scoring functions, the study employs Monte Carlo simulations and theoretical analysis to demonstrate that time-varying volatility and asymmetric loss jointly induce severe skewness in test statistics, distorting both standard and bootstrap critical values and drastically reducing statistical power—leading to frequent Type III errors. This work is the first to systematically characterize this dual-driver mechanism, rigorously delineates the validity boundaries of DM and MCS for tail-risk evaluation, and proposes diagnostic criteria to flag invalid applications alongside concrete directions for methodological improvement.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success

Abstract

This paper provides comprehensive simulation results on the finite sample properties of the Diebold-Mariano (DM) test by Diebold and Mariano (1995) and the model confidence set (MCS) testing procedure by Hansen et al. (2011) applied to the asymmetric loss functions specific to financial tail risk forecasts, such as Value-at-Risk (VaR) and Expected Shortfall (ES). We focus on statistical loss functions that are strictly consistent in the sense of Gneiting (2011a). We find that the tests show little power against models that underestimate the tail risk at the most extreme quantile levels, while the finite sample properties generally improve with the quantile level and the out-of-sample size. For the small quantile levels and out-of-sample sizes of up to two years, we observe heavily skewed test statistics and non-negligible type III errors, which implies that researchers should be cautious about using standard normal or bootstrapped critical values. We demonstrate both empirically and theoretically how these unfavorable finite sample results relate to the asymmetric loss functions and the time varying volatility inherent in financial return data.

Problem

• Evaluating Diebold-Mariano and MCS tests for financial tail risk forecasts
• Assessing test power for extreme quantile levels in risk models
• Analyzing skewed statistics and errors in small sample scenarios

Innovation

• Uses Diebold-Mariano test for tail risk
• Applies Model Confidence Set procedure
• Focuses on asymmetric loss functions