xai_evals : A Framework for Evaluating Post-Hoc Local Explanation Methods

2025-02-05

AI Summary

To address the lack of rigorous evaluation criteria for post-hoc explanation methods for black-box models, this paper introduces the first open-source, multimodal evaluation framework supporting both tabular and image data. The framework unifies three core dimensions—faithfulness, sensitivity, and robustness—into a reproducible, systematic benchmarking protocol. It integrates major explanation methods including SHAP, LIME, Grad-CAM, Integrated Gradients, and Backpropagation-based Trace (Backtrace), and supports models implemented in PyTorch and TensorFlow, as well as user-defined explainers. Extensive experiments on UCI tabular benchmarks and ImageNet demonstrate substantial performance variation across methods, underscoring the necessity of standardized evaluation. Our framework significantly improves the efficiency, comparability, and reproducibility of explanation credibility assessment. The implementation is publicly released and has been widely adopted by the research community.

Technology Category

• Computer Vision: Interpretability, Explainability, and Transparency
• Machine Learning: Calibration & Uncertainty Quantification
• Humans and AI: Explainable AI (XAI) for Human Understanding

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

The growing complexity of machine learning and deep learning models has led to an increased reliance on opaque"black box"systems, making it difficult to understand the rationale behind predictions. This lack of transparency is particularly challenging in high-stakes applications where interpretability is as important as accuracy. Post-hoc explanation methods are commonly used to interpret these models, but they are seldom rigorously evaluated, raising concerns about their reliability. The Python package xai_evals addresses this by providing a comprehensive framework for generating, benchmarking, and evaluating explanation methods across both tabular and image data modalities. It integrates popular techniques like SHAP, LIME, Grad-CAM, Integrated Gradients (IG), and Backtrace, while supporting evaluation metrics such as faithfulness, sensitivity, and robustness. xai_evals enhances the interpretability of machine learning models, fostering transparency and trust in AI systems. The library is open-sourced at https://pypi.org/project/xai-evals/ .

Problem

• Evaluates post-hoc explanation methods reliability
• Enhances interpretability of machine learning models
• Supports benchmarking across tabular and image data

Innovation

• Framework for evaluating explanation methods
• Supports SHAP, LIME, Grad-CAM techniques
• Measures faithfulness, sensitivity, robustness metrics