forecast evaluation

This work addresses the limitation of existing safety-critical systems, which typically evaluate only predictive accuracy while lacking rigorous validation of the overall calibration of predicted probability distributions. To bridge this gap, the authors propose a modular calibration testing framework that decouples the calibration process into four interchangeable components: data model, scoring rule, hypothesis formulation, and statistical test procedure. Built upon formal statistical hypothesis testing, the framework provides a single accept/reject decision for the entire predictive distribution. Crucially, it rejects only overly confident predictions while tolerating reasonable deviations, thereby balancing practicality with flexibility. Empirical evaluations on weather forecasting and robotic pose estimation tasks demonstrate that the framework effectively supports reliable deployment in safety-critical applications.

Statistical reliability assessment of the Pareto front in multi-objective optimization remains challenging, particularly for non-deterministic systems requiring rigorous, verifiable guarantees. Method: This paper introduces the first lightweight statistical model checking framework supporting multi-objective Pareto queries. It integrates Monte Carlo simulation with incremental policy sampling, a progressive convergence mechanism, and three efficient heuristics to compute two-sided confidence intervals for the Pareto front under limited sampling budgets. The approach unifies statistical hypothesis testing with online learning, ensuring almost-sure convergence. Results: Evaluated on the Modest/modes platform, the method achieves high-fidelity approximation of the true Pareto front at significantly lower computational cost than existing approaches, while delivering statistically sound, verifiable confidence guarantees. It substantially enhances both the rigor and practicality of multi-attribute trade-off analysis in stochastic systems.

This work proposes a Python-driven workflow built upon the Storm model checker to address the challenges of efficient probabilistic model checking and strategy synthesis for systems with uncertainty, such as Markov decision processes (MDPs). The approach seamlessly integrates high-performance verification engines with the rich Python ecosystem, supporting state-of-the-art uncertain models including standard MDPs, interval MDPs, and partially observable MDPs (POMDPs). It enables an end-to-end pipeline encompassing modeling, property verification, and policy extraction. By bridging formal verification tools with flexible scripting capabilities, the framework significantly enhances the usability, scalability, and deployment adaptability of probabilistic model checking, facilitating rapid customization and integration of novel algorithms in complex real-world applications.

This paper addresses the challenge of dynamically aligning probabilistic system models with their actual runtime behavior. Methodologically, it proposes a lightweight, real-time alignment monitoring framework featuring an online-computable alignment score, novel differential alignment monitoring (to detect local misalignment trends), and weighted alignment monitoring (to support task-specific customization and model comparison). The monitor is built upon sequential prediction, integrating probabilistic forecasts, distributional similarity metrics (e.g., Wasserstein distance), and high-confidence interval estimation for runtime assessment. Experiments on the PRISM benchmark demonstrate that the monitor incurs low memory overhead, responds rapidly, and effectively detects model–reality misalignment with high accuracy and strong real-time performance. This work establishes a new paradigm for trustworthy verification of probabilistic systems.

State estimators’ self-assessed uncertainties (e.g., covariance matrices) are often unreliable due to noise or system model mismatch, compromising downstream decision-making safety. To address this, we propose a unified multi-metric credibility assessment framework integrating normalized estimation error squared (NEES), noncredibility index (NUI), negative log-likelihood (NLL), and energy score (ES), augmented by a novel energy-distance-based positional test. Leveraging the asymmetric sensitivity of NLL and ES—NLL to overconfident (underdispersed) covariances and ES to systematic estimation biases—we enable distinguishable diagnosis of distinct model misspecification types. Evaluated across six canonical mismatch scenarios, our framework achieves 80–100% classification accuracy, substantially outperforming single-metric baselines. It provides an interpretable, reproducible paradigm for uncertainty calibration and fault attribution in state estimation.

This study addresses the challenge of effectively validating input model specifications in digital twin simulations, where conventional approaches—relying solely on marginal output distributions—often fail to detect misspecified joint input models. To overcome this limitation, the authors propose a novel statistical validation framework based on sub-trajectory conditioning. By repeatedly restarting simulations from observed system states while conditioning on subsets of random inputs, the method constructs conditional output distributions that enable goodness-of-fit testing of the full joint input model. This approach innovatively transcends the constraints of marginal validation and is complemented by diagnostic tools to pinpoint specific input sources responsible for detected discrepancies. Empirical evaluations on M/M/1 and tandem queueing systems demonstrate the framework’s heightened sensitivity and effectiveness, successfully identifying input model misspecifications that traditional methods overlook.

This study addresses the lack of a systematic framework for identifying critical input variables and conducting sensitivity analysis under uncertainty in complex simulations, particularly in military decision-making contexts. The authors propose a unified sensitivity analysis framework that integrates local and global methods—including variance-based, derivative-based, screening, and uncertainty quantification techniques—and strategically maps these approaches to specific decision objectives such as factor prioritization, fixing, variance reduction, and mapping. Innovatively, the framework introduces a “sensitivity audit” mechanism to enhance traceability of model assumptions and promote responsible model usage. By providing a structured guide for high-dimensional, complex simulation systems, this work significantly improves model interpretability, transparency, and the credibility of decisions derived from such models.

This study addresses the insufficient quantification of uncertainty in numerical weather prediction by introducing, for the first time, multiple conformal prediction (CP) methods—including standard CP, normalized CP, and conformal quantile regression—into a one-dimensional shallow-water model data assimilation framework. These methods are integrated with the ensemble Kalman filter to produce prediction intervals endowed with finite-sample theoretical guarantees. Systematic evaluation using metrics such as average coverage, interval width, upper and lower tail miss rates, and interval score demonstrates that CP effectively characterizes forecast uncertainty and complements traditional ensemble-based approaches within the data assimilation cycle. The work establishes a novel paradigm and empirical foundation for uncertainty quantification that synergistically combines machine learning with physics-driven modeling.

This study addresses the limitations of existing SysML verification approaches, which are often tool-dependent and restricted to performance properties, lacking support for automated validation of behavioral and interface requirements. To overcome these shortcomings, this work proposes a tool-agnostic, automated verification workflow driven by SysML test cases, integrating UML Testing Profile and behavioral diagram constructs to enable unified validation of multidimensional attributes—including behavior, timing, and state responses. The methodology was developed through a mixed-methods research strategy combining literature review and stakeholder interviews, and its efficacy was empirically validated across two independent SysML toolchains. The approach not only transcends the constraints of conventional parametric methods but also enables automatic traceability of verification results back to the original model elements.

This study addresses the reliability of probabilistic uncertainty quantification (UQ) in software defect prediction, particularly its ability to reflect model performance and calibration—especially in cross-project settings, where systematic validation remains lacking. Through a large-scale empirical analysis of 16 classifiers across 36 within-project and 32 cross-project datasets, the work examines the relationships between five UQ metrics and six performance measures alongside three calibration metrics. It reveals, for the first time, a strong context dependency: within-project, UQ correlates strongly with false positive rate and AUC, but these correlations substantially weaken or even reverse in cross-project scenarios. Notably, high-performing models can still exhibit severe miscalibration. These findings indicate that UQ signals are not directly transferable and must be evaluated independently relative to specific objectives, using multidimensional calibration assessments.