🤖 AI Summary
This study addresses the degradation of explanation fidelity caused by data drift, wherein legacy explanations fail to effectively monitor updated model behavior. To investigate this, we employ ExIFFI to generate local explanations and integrate conditional drift detection with multi-level intervention evaluation, systematically examining how covariate shift affects explanation faithfulness. Our work provides the first empirical evidence that structural stability does not entail functional faithfulness, demonstrating that explanations must be independently monitored and bound to specific models. Furthermore, we show that although pre-retraining explanations retain relevance after model updates, their fidelity is significantly inferior to that of newly generated explanations, while no systematic widening of this discrepancy is observed.
📝 Abstract
Model performance monitoring is a standard practice in machine learning deployments. Detection performance is tracked continuously, and model decay is expected as the relationship between the feature and target variables degrades, a phenomenon known as concept drift. Explanation fidelity, however, is rarely monitored with the same discipline, even in domains such as financial systems, healthcare, and other regulated environments where explanations are required for governance purposes. This paper investigates whether explanations can decay under data drift, even when the feature-target relationship remains stable, and whether explanations produced before drift occurs remain faithful to the decisions of the model that replaces them. Using the IEEE-CIS Transaction Fraud Detection dataset, we find statistically significant covariate shift but no statistically significant evidence of concept drift under the implemented conditional-drift tests, thereby providing an empirical setting in which input distributional change can be studied separately from detectable changes in the feature-target relationship. Local explanations are generated with ExIFFI and evaluated at three levels: path validity, structural behaviour, and fidelity under controlled intervention. Results show that while prior explanations retain substantial decision relevance to a retrained model, they are consistently less faithful than newly generated explanations, with no evidence of a systematically widening gap across the evaluated windows. The study shows that explanation fidelity requires its own monitoring, that structural stability of explanations does not guarantee functional fidelity, and that explanations should be treated as artifacts tied to the model that produced them.