🤖 AI Summary
In text summarization, diverse XAI methods frequently yield contradictory attributions for the same model output—a phenomenon termed the “disagreement problem”—which critically undermines explanation credibility and AI accountability. This work presents the first systematic empirical investigation of this issue in summarization. We propose Regionalized eXplainable AI (RXAI), a novel framework that abandons the global attribution assumption by partitioning the source document into semantically coherent segments and performing attribution independently per segment. RXAI integrates Sentence-BERT embeddings, hierarchical clustering for segmentation, and multiple XAI techniques—including Integrated Gradients and LIME—augmented with interactive sentence-level visualization. Evaluated on XSum and CNN/Daily Mail, RXAI significantly reduces cross-method attribution disagreement and improves local attribution consistency. Our approach establishes a new paradigm for fine-grained, reliable, and verifiable summarization explanations.
📝 Abstract
Explainable Artificial Intelligence (XAI) methods in text summarization are essential for understanding the model behavior and fostering trust in model-generated summaries. Despite the effectiveness of XAI methods, recent studies have highlighted a key challenge in this area known as the"disagreement problem". This problem occurs when different XAI methods yield conflicting explanations for the same model outcome. Such discrepancies raise concerns about the consistency of explanations and reduce confidence in model interpretations, which is crucial for secure and accountable AI applications. This work is among the first to empirically investigate the disagreement problem in text summarization, demonstrating that such discrepancies are widespread in state-of-the-art summarization models. To address this gap, we propose Regional Explainable AI (RXAI) a novel segmentation-based approach, where each article is divided into smaller, coherent segments using sentence transformers and clustering. We use XAI methods on text segments to create localized explanations that help reduce disagreement between different XAI methods, thereby enhancing the trustworthiness of AI-generated summaries. Our results illustrate that the localized explanations are more consistent than full-text explanations. The proposed approach is validated using two benchmark summarization datasets, Extreme summarization (Xsum) and CNN/Daily Mail, indicating a substantial decrease in disagreement. Additionally, the interactive JavaScript visualization tool is developed to facilitate easy, color-coded exploration of attribution scores at the sentence level, enhancing user comprehension of model explanations.