🤖 AI Summary
This work addresses the limited interpretability of internal mechanisms in reinforcement learning—such as value estimation, policy optimization, and their interaction with temporal difference (TD) signals—by proposing the first systematic, multi-perspective framework for visualizing loss landscapes. The approach integrates the geometric structure of value functions, policy optimization trajectories, TD error dynamics, and state-driven regions through techniques including 3D loss surface reconstruction, policy landscape visualization under a frozen critic, joint trajectories of time–Bellman error–policy weights, and state–TD mappings. Applied to the ADHDP algorithm for spacecraft attitude control, the framework enables comparative analysis of multiple variants, revealing how training stabilizers and target update mechanisms reshape the optimization landscape and influence learning stability. This study establishes a new paradigm for interpretable reinforcement learning and provides actionable insights for algorithm design.
📝 Abstract
Reinforcement learning algorithms have been widely used in dynamic and control systems. However, interpreting their internal learning behavior remains a challenge. In the authors' previous work, a critic match loss landscape visualization method was proposed to study critic training. This study extends that method into a framework which provides a multi-perspective view of the learning dynamics, clarifying how value estimation, policy optimization, and temporal-difference (TD) signals interact during training. The proposed framework includes four complementary components; a three-dimensional reconstruction of the critic match loss surface that shows how TD targets shape the optimization geometry; an actor loss landscape under a frozen critic that reveals how the policy exploits that geometry; a trajectory combining time, Bellman error, and policy weights that indicates how updates move across the surface; and a state-TD map that identifies the state regions that drive those updates. The Action-Dependent Heuristic Dynamic Programming (ADHDP) algorithm for spacecraft attitude control is used as a case study. The framework is applied to compare several ADHDP variants and shows how training stabilizers and target updates change the optimization landscape and affect learning stability. Therefore, the proposed framework provides a systematic and interpretable tool for analyzing reinforcement learning behavior across algorithmic designs.