Revisit Time Series Classification Benchmark: The Impact of Temporal Information for Classification

2025-03-26

AI Summary

Many datasets in the UCR time series classification benchmark exhibit low sensitivity to temporal information, biasing current evaluations against models’ sequential modeling capabilities. Method: The authors propose the first systematic quantification of temporal information contribution per dataset, introducing a permutation-test-based framework for assessing temporal sensitivity and constructing UCR Augmented—the first benchmark explicitly designed to evaluate temporal-awareness. This benchmark decouples temporal dynamics from static features to enable fair, interpretable comparisons of models’ temporal modeling capacity. Contribution/Results: Evaluation on UCR Augmented substantially reorders model rankings: lightweight methods (e.g., ROCKET, TSF) excel on low-temporal-dependency tasks, while strong sequential models (e.g., LSTM, InceptionTime) achieve marked performance gains on high-sensitivity tasks. These results validate UCR Augmented’s effectiveness and discriminative power in revealing true temporal modeling competence.

Technology Category

Machine Learning: Time-Series/Data StreamsKnowledge Representation and Reasoning: Geometric, Spatial, and Temporal ReasoningPlanning, Routing, and Scheduling: Temporal Planning

Application Category

Search and Retrieval-Augmented AI: Web evaluation methodologies and metricsGraph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphsUser Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Time series classification is usually regarded as a distinct task from tabular data classification due to the importance of temporal information. However, in this paper, by performing permutation tests that disrupt temporal information on the UCR time series classification archive, the most widely used benchmark for time series classification, we identify a significant proportion of datasets where temporal information has little to no impact on classification. Many of these datasets are tabular in nature or rely mainly on tabular features, leading to potentially biased evaluations of time series classifiers focused on temporal information. To address this, we propose UCR Augmented, a benchmark based on the UCR time series classification archive designed to evaluate classifiers' ability to extract and utilize temporal information. Testing classifiers from seven categories on this benchmark revealed notable shifts in performance rankings. Some previously overlooked approaches perform well, while others see their performance decline significantly when temporal information is crucial. UCR Augmented provides a more robust framework for assessing time series classifiers, ensuring fairer evaluations. Our code is available at https://github.com/YunruiZhang/Revisit-Time-Series-Classification-Benchmark.

Problem

• Assess temporal information impact on time series classification
• Identify datasets where temporal info is insignificant for classification
• Propose UCR Augmented for robust classifier evaluation

Innovation

• Permutation tests disrupt temporal information
• UCR Augmented evaluates temporal information usage
• Performance rankings shift with temporal importance