Why do explanations fail? A typology and discussion on failures in XAI

2024-05-22

arXiv.org

AI Summary

Why do XAI explanations fail? Prior work often attributes failures in isolation to either technical limitations or user misunderstandings. This paper argues that XAI failure is fundamentally systemic—arising from the interplay of system- and user-level factors. To address this, we propose the first dual-dimensional typology distinguishing *system-specific failures* (e.g., model unreliability, explanation inconsistency) from *user-specific failures* (e.g., cognitive biases, domain knowledge gaps). Our methodology integrates systematic literature review, conceptual analysis, and interdisciplinary critical reflection grounded in human factors engineering and explainability theory. This framework transcends fragmented problem analyses by enabling the first holistic, multi-layered modeling of XAI failure, clarifying causal structures and identifying critical research gaps. It provides both a theoretical foundation and actionable guidance for designing robust, trustworthy, and user-adapted explanation methods.

Technology Category

• Humans and AI: Explainable AI (XAI) for Human Understanding
• Reasoning under Uncertainty: Causality
• Philosophy and Ethics of AI: Accountability, Interpretability & Explainability

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

As Machine Learning (ML) models achieve unprecedented levels of performance, the XAI domain aims at making these models understandable by presenting end-users with intelligible explanations. Yet, some existing XAI approaches fail to meet expectations: several issues have been reported in the literature, generally pointing out either technical limitations or misinterpretations by users. In this paper, we argue that the resulting harms arise from a complex overlap of multiple failures in XAI, which existing ad-hoc studies fail to capture. This work therefore advocates for a holistic perspective, presenting a systematic investigation of limitations of current XAI methods and their impact on the interpretation of explanations. By distinguishing between system-specific and user-specific failures, we propose a typological framework that helps revealing the nuanced complexities of explanation failures. Leveraging this typology, we also discuss some research directions to help AI practitioners better understand the limitations of XAI systems and enhance the quality of ML explanations.

Problem

• Investigating failures in explainable AI systems and user interpretations
• Proposing a typological framework for system-specific and user-specific failures
• Discussing research directions to enhance XAI limitations understanding

Innovation

• Proposes typological framework for XAI failures
• Distinguishes system-specific and user-specific failures
• Advocates holistic perspective for XAI limitations