Class-Dependent Perturbation Effects in Evaluating Time Series Attributions

2025-02-24

AI Summary

In time series classification, perturbation-based feature attribution evaluation suffers from significant class-dependent perturbation effects: the same perturbation strategy exhibits markedly varying sensitivity across classes, causing evaluation results to be confounded by inherent classifier biases and thus failing to reflect the true quality of attribution methods. This paper is the first to systematically identify and model this class-dependent perturbation effect, proposing a novel evaluation framework incorporating a class-aware penalty term. Through empirical analysis across multiple datasets, models, and perturbation strategies, we demonstrate that mainstream perturbation metrics—such as AOPC and IROF—are highly susceptible to class-wise bias. Our framework effectively calibrates such evaluation bias, enhancing fairness and reliability in comparative attribution assessment. The proposed approach establishes a more robust foundation for evaluating explainable AI in time series analysis.

Technology Category

• Machine Learning: Evaluation and Analysis
• Computer Vision: Adversarial Attacks & Robustness
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

As machine learning models become increasingly prevalent in time series applications, Explainable Artificial Intelligence (XAI) methods are essential for understanding their predictions. Within XAI, feature attribution methods aim to identify which input features contributed the most to a model's prediction, with their evaluation typically relying on perturbation-based metrics. Through empirical analysis across multiple datasets, model architectures, and perturbation strategies, we identify important class-dependent effects in these metrics: they show varying effectiveness across classes, achieving strong results for some while remaining less sensitive to others. In particular, we find that the most effective perturbation strategies often demonstrate the most pronounced class differences. Our analysis suggests that these effects arise from the learned biases of classifiers, indicating that perturbation-based evaluation may reflect specific model behaviors rather than intrinsic attribution quality. We propose an evaluation framework with a class-aware penalty term to help assess and account for these effects in evaluating feature attributions. Although our analysis focuses on time series classification, these class-dependent effects likely extend to other structured data domains where perturbation-based evaluation is common.

Problem

• Evaluate class-dependent effects in feature attribution
• Identify bias in perturbation-based metrics
• Propose framework for class-aware attribution evaluation

Innovation

• Class-dependent perturbation effects analysis
• Class-aware penalty term framework
• Evaluation across multiple datasets strategies