Defining Explainable AI for Requirements Analysis

2026-02-22

AI Summary

This work addresses the critical challenge of eliciting and formalizing explanation requirements for Explainable Artificial Intelligence (XAI) across diverse application contexts to foster user trust. The authors propose a three-dimensional taxonomy—encompassing Source, Depth, and Scope—to systematically characterize the heterogeneous interpretability demands of different use cases and align them with the explanatory capabilities of machine learning models. Notably, this framework is agnostic to specific XAI algorithms and establishes, for the first time, a structured bridge between requirements engineering and XAI. By providing a principled foundation for articulating what constitutes a “good” explanation in context, the framework offers both theoretical grounding and practical guidance for the design, selection, and evaluation of XAI systems.

Technology Category

Humans and AI: Explainable AI (XAI) for Human UnderstandingMachine Learning: Transparent, Interpretable, Explainable MLPhilosophy and Ethics of AI: Accountability, Interpretability & Explainability

Application Category

User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalizationSemantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactionsSearch and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Explainable Artificial Intelligence (XAI) has become popular in the last few years. The Artificial Intelligence (AI) community in general, and the Machine Learning (ML) community in particular, is coming to the realisation that in many applications, for AI to be trusted, it must not only demonstrate good performance in its decisionmaking, but it also must explain these decisions and convince us that it is making the decisions for the right reasons. However, different applications have different requirements on the information required of the underlying AI system in order to convince us that it is worthy of our trust. How do we define these requirements? In this paper, we present three dimensions for categorising the explanatory requirements of different applications. These are Source, Depth and Scope. We focus on the problem of matching up the explanatory requirements of different applications with the capabilities of underlying ML techniques to provide them. We deliberately avoid including aspects of explanation that are already well-covered by the existing literature and we focus our discussion on ML although the principles apply to AI more broadly.

Problem

• Explainable AI
• Requirements Analysis
• Machine Learning
• Trust
• Explanation

Innovation

• Explainable AI
• Requirements Analysis
• Explanation Dimensions
• Machine Learning
• Trustworthy AI