ForeComp: An R Package for Comparing Predictive Accuracy Using Fixed-Smoothing Asymptotics

2026-03-08

AI Summary

This study addresses the sensitivity of bandwidth selection in loss-differential tests for comparing predictive accuracy, which can critically affect both the size and power of such tests. To enhance robustness, the authors propose a fixed-smoothing asymptotic approach and implement it in the R package ForeComp, offering a unified interface for Diebold–Mariano-type tests. A novel Plot Tradeoff diagnostic is introduced to visually illustrate the trade-off between bandwidth sensitivity and test power, thereby improving transparency in bandwidth choice and reliability of inference. Extensive Monte Carlo simulations and empirical analysis using professional forecasters’ survey data demonstrate that the proposed method substantially improves the finite-sample statistical performance and practical utility of predictive accuracy comparisons.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Search and Optimization: Metareasoning and Metaheuristics
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

We introduce ForeComp, an R package for comparing predictive accuracy using Diebold-Mariano type tests of equal predictive ability with standard and fixed smoothing inference. The package provides a common interface for loss differential based testing and includes Plot Tradeoff, a visual diagnostic for bandwidth sensitivity and the size-power tradeoff. We illustrate the toolkit with Survey of Professional Forecasters applications and Monte Carlo evidence on finite-sample performance.

Problem

• predictive accuracy
• Diebold-Mariano test
• fixed-smoothing asymptotics
• bandwidth sensitivity
• size-power tradeoff

Innovation

• fixed-smoothing asymptotics
• Diebold-Mariano test
• predictive accuracy comparison
• bandwidth sensitivity
• size-power tradeoff