STACHE: Local Black-Box Explanations for Reinforcement Learning Policies

2025-12-10

AI Summary

Reinforcement learning agents often exhibit uninterpretable actions and unreliable behavior in sparse-reward and safety-critical settings. Method: This paper proposes a local black-box explanation framework for discrete Markov games. It introduces, for the first time, a composite explanation structure combining *robust regions* and *minimal counterfactuals*. Leveraging a factorized state space, we design an exact search algorithm that avoids fidelity loss from surrogate models. The framework integrates robust region computation, minimal counterfactual generation, and a black-box policy query interface. Results: Experiments in Gymnasium demonstrate that the framework efficiently identifies critical sensitivity boundaries of policy actions, clearly characterizes the logical evolution of policies—from unstable to robust—during training, and significantly improves explanation accuracy and verifiability.

Technology Category

• Multiagent Systems: Adversarial Agents
• Reasoning under Uncertainty: Sequential Decision Making
• Machine Learning: Adversarial Learning & Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization

Abstract

Reinforcement learning agents often behave unexpectedly in sparse-reward or safety-critical environments, creating a strong need for reliable debugging and verification tools. In this paper, we propose STACHE, a comprehensive framework for generating local, black-box explanations for an agent's specific action within discrete Markov games. Our method produces a Composite Explanation consisting of two complementary components: (1) a Robustness Region, the connected neighborhood of states where the agent's action remains invariant, and (2) Minimal Counterfactuals, the smallest state perturbations required to alter that decision. By exploiting the structure of factored state spaces, we introduce an exact, search-based algorithm that circumvents the fidelity gaps of surrogate models. Empirical validation on Gymnasium environments demonstrates that our framework not only explains policy actions, but also effectively captures the evolution of policy logic during training - from erratic, unstable behavior to optimized, robust strategies - providing actionable insights into agent sensitivity and decision boundaries.

Problem

• Explains reinforcement learning agent actions in discrete Markov games
• Generates local black-box explanations via robustness regions and counterfactuals
• Captures policy logic evolution and decision boundaries during training

Innovation

• Generates composite explanations with robustness regions and counterfactuals
• Uses exact search-based algorithm for factored state spaces
• Captures policy evolution from unstable to optimized strategies