Interactive Explanations for Reinforcement-Learning Agents

2025-04-07

AI Summary

To address the opacity of decision-making in reinforcement learning (RL) agents, this paper proposes ASQ-IT, an interactive explainability framework that introduces sociological conversational explanation paradigms into eXplainable RL (XRL) for the first time. Methodologically, it formalizes natural-language temporal queries using Linear Temporal Logic over finite traces (LTLf), integrates finite-trace automata to construct verifiable and interactive explanation models, and designs a video-behavior-segment retrieval algorithm alongside a query–interface mapping mechanism. User studies demonstrate that non-expert users rapidly acquire proficiency in LTLf-style querying, achieving a 37% improvement in detecting agent anomalies and reducing average diagnostic time by 52%. This work establishes the first queryable XRL system supporting natural-language-driven, temporally aware, and human-in-the-loop debugging.

Technology Category

• Humans and AI: Explainable AI (XAI) for Human Understanding
• Machine Learning: Reinforcement Learning
• Multiagent Systems: Adversarial Agents

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

As reinforcement learning methods increasingly amass accomplishments, the need for comprehending their solutions becomes more crucial. Most explainable reinforcement learning (XRL) methods generate a static explanation depicting their developers' intuition of what should be explained and how. In contrast, literature from the social sciences proposes that meaningful explanations are structured as a dialog between the explainer and the explainee, suggesting a more active role for the user and her communication with the agent. In this paper, we present ASQ-IT -- an interactive explanation system that presents video clips of the agent acting in its environment based on queries given by the user that describe temporal properties of behaviors of interest. Our approach is based on formal methods: queries in ASQ-IT's user interface map to a fragment of Linear Temporal Logic over finite traces (LTLf), which we developed, and our algorithm for query processing is based on automata theory. User studies show that end-users can understand and formulate queries in ASQ-IT and that using ASQ-IT assists users in identifying faulty agent behaviors.

Problem

• Enhancing comprehension of reinforcement learning solutions
• Providing interactive explanations via user-agent dialogue
• Identifying faulty agent behaviors through formal query methods

Innovation

• Interactive explanation system with user queries
• Formal methods using Linear Temporal Logic
• Automata theory for query processing