🤖 AI Summary
This work addresses the lack of direct metrics and control mechanisms for critic model complexity in conventional actor-critic reinforcement learning. It introduces spectral effective rank entropy—a novel measure derived from singular value decomposition—to quantify the complexity of critic weight matrices. The relationship between this complexity metric and training dynamics is empirically analyzed within TD3 and PPO algorithms. Building on these insights, the study proposes a spectral entropy regularization term to actively regulate critic complexity. Experimental results demonstrate that critic complexity can be reliably measured and effectively controlled, with its impact on performance exhibiting strong task dependence. These findings reveal heterogeneous interactions among model complexity, algorithmic design, task characteristics, and hyperparameter choices.
📝 Abstract
Actor-critic methods depend on learned critics, but critic quality is often evaluated only indirectly through return, temporal-difference error, or value loss. Critic complexity is introduced as an additional diagnostic and intervention dimension for actor-critic reinforcement learning. The analysis uses spectral effective-rank entropy, a rank-like summary of the singular-value distributions of critic weight matrices, to assess critic model complexity. Across TD3 and PPO experiments, critic complexity is tracked together with return and Monte Carlo value-estimation bias. The results show that critic complexity is measurable throughout training and is systematically associated with training behavior, while also making clear that the relationship is heterogeneous across algorithms, tasks, and hyperparameters. A direct complexity-control intervention is then evaluated by adding a spectral-entropy penalty to the critic loss. This intervention reliably changes the targeted spectral quantity, demonstrating that critic complexity can be controlled rather than only observed. Return effects are treated as task-dependent evidence rather than as a general performance claim, because overall complexity-control results vary.