A Survey of Explainable Reinforcement Learning: Targets, Methods and Needs

2025-07-16

AI Summary

To address the opacity of deep neural network–based decision-making in reinforcement learning (RL) agents, this paper presents a systematic survey of explainable RL (XRL). We propose a dual-axis taxonomy—“what to explain” (target dimension: e.g., policy, value function, trajectory) and “how to explain” (method dimension: e.g., surrogate modeling, attention visualization, counterfactual generation)—to structurally categorize over 250 XRL studies. This framework unifies disparate classification logics in prior work and exposes critical gaps in adaptability to dynamic environments, human interpretability, and real-time explanation capability. Our analysis identifies three pressing research directions: (1) multi-granularity explanation integration, (2) human-in-the-loop evaluation mechanisms, and (3) lightweight, trustworthy explanation paradigms tailored to real-world RL applications—including robotic control and autonomous driving. The survey thus provides both theoretical foundations and practical guidelines for advancing XRL research and deployment.

Technology Category

• Humans and AI: Explainable AI (XAI) for Human Understanding
• Machine Learning: Reinforcement Learning
• Philosophy and Ethics of AI: Accountability, Interpretability & Explainability

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

The success of recent Artificial Intelligence (AI) models has been accompanied by the opacity of their internal mechanisms, due notably to the use of deep neural networks. In order to understand these internal mechanisms and explain the output of these AI models, a set of methods have been proposed, grouped under the domain of eXplainable AI (XAI). This paper focuses on a sub-domain of XAI, called eXplainable Reinforcement Learning (XRL), which aims to explain the actions of an agent that has learned by reinforcement learning. We propose an intuitive taxonomy based on two questions "What" and "How". The first question focuses on the target that the method explains, while the second relates to the way the explanation is provided. We use this taxonomy to provide a state-of-the-art review of over 250 papers. In addition, we present a set of domains close to XRL, which we believe should get attention from the community. Finally, we identify some needs for the field of XRL.

Problem

• Surveying explainable methods in reinforcement learning (XRL).
• Classifying XRL methods by 'What' and 'How' questions.
• Reviewing 250+ papers to identify XRL trends and needs.

Innovation

• Surveying explainable reinforcement learning methods
• Taxonomy based on 'What' and 'How' questions
• Reviewing over 250 XRL papers comprehensively