BITS: Rethinking Fair and Comprehensive Evaluation for Irregular Time Series Forecasting

2026-09-27

AI Summary

Irregular time series forecasting lacks a unified benchmark for fair comparison due to limited datasets, inconsistent evaluation protocols, and reliance on single error metrics. To address this, we propose BITS, a standardized benchmark comprising eleven multi-domain datasets with explicit characterization of irregularity features such as missing rates. We construct a unified pipeline encompassing data preprocessing, model integration, and evaluation that accommodates conventional methods, irregular-aware approaches, and foundation models. Furthermore, we establish a multidimensional quantitative evaluation framework by integrating both error-based and non-error-based metrics. Experiments demonstrate that model performance varies significantly with data irregularity, no single strategy consistently dominates, and different metrics yield divergent rankings, thereby validating the necessity of multidimensional evaluation.

Abstract

Despite recent progress in irregular time series forecasting, the field still lacks a unified benchmark for fair and comprehensive evaluation. Existing evaluations are often conducted on a limited set of datasets with inconsistent experimental protocols and predominantly error-based metrics, rendering it difficult to compare and assess methods fairly and comprehensively across diverse settings. To eliminate these limitations and accelerate progress, we propose BITS, a standardized, reproducible, and extensible benchmark for advancing research on irregular time series forecasting. BITS covers eleven datasets from nine domains with diverse irregularity characteristics, and it characterizes the datasets according to their missing rate, missing pattern complexity, sampling irregularity, and skewness. Further, it offers a unified pipeline for data preprocessing, model integration and evaluation, and reporting. It accommodates regular and irregular time series forecasting methods, including time series foundation models, under consistent settings, incorporating both error-based and non-error-based evaluation metrics. Findings include that method performance varies substantially across irregularity characteristics, with no single modeling strategy consistently dominating. We also find that using error-based or non-error-based metrics can yield different model rankings, highlighting the need for multi-dimensional evaluation. The code can be found at https://anonymous.4open.science/r/BITS-8F2E/.

Problem

• irregular time series forecasting
• benchmark
• fair evaluation
• comprehensive evaluation

Innovation

• Irregular Time Series Forecasting
• Unified Benchmark
• Time Series Foundation Models
• Multi-dimensional Evaluation
• Standardized Pipeline