Statistical Model Checking Beyond Means: Quantiles, CVaR, and the DKW Inequality (extended version)

2025-09-15

AI Summary

Traditional statistical model checking (SMC) is limited to mean-type statistics—such as expected reward and probability—and thus struggles to reliably estimate distribution-sensitive risk measures, including quantiles, conditional value-at-risk (CVaR), and entropy-based risk. This work introduces, for the first time, the Dvoretzky–Kiefer–Wolfowitz–Massart (DKW) inequality into SMC, enabling non-asymptotic confidence bands around the empirical cumulative distribution function (ECDF) to perform rigorous, distribution-wide inference over system trajectories. The resulting framework supports statistical verification of key risk metrics—quantiles, CVaR, and entropy risk—with formal guarantees. It is efficiently implemented in the MODEST Toolset’s modes simulator. Experimental evaluation across multiple quantitative verification benchmarks demonstrates high accuracy and strong robustness, significantly extending the risk modeling capabilities and expressive power of SMC for reliability assessment.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Stochastic Optimization
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Security and Privacy: Large-scale security measurements
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Statistical model checking (SMC) randomly samples probabilistic models to approximate quantities of interest with statistical error guarantees. It is traditionally used to estimate probabilities and expected rewards, i.e. means of different random variables on paths. In this paper, we develop methods using the Dvoretzky-Kiefer-Wolfowitz-Massart inequality (DKW) to extend SMC beyond means to compute quantities such as quantiles, conditional value-at-risk, and entropic risk. The DKW provides confidence bounds on the random variable's entire cumulative distribution function, a much more versatile guarantee compared to the statistical methods prevalent in SMC today. We have implemented support for computing new quantities via the DKW in the 'modes' simulator of the Modest Toolset. We highlight the implementation and its versatility on benchmarks from the quantitative verification literature.

Problem

• Extends statistical model checking beyond mean estimation
• Computes quantiles and risk measures using DKW inequality
• Provides confidence bounds on full cumulative distribution functions

Innovation

• Extends SMC using DKW inequality
• Computes quantiles and risk measures
• Provides confidence bounds on distribution