🤖 AI Summary
Existing model-free reinforcement learning lacks a general-purpose mechanism for enforcing policy constraints; prior approaches support only specific constraint types (e.g., value or density constraints).
Method: We propose DualCRL, a unified primal-dual framework that models diverse behavioral constraints—including action density bounds and state-action transition costs—as learnable reward shaping terms, seamlessly integrating with both value-based and actor-critic paradigms.
Contribution/Results: DualCRL establishes, for the first time, the formal equivalence between dual variables and reward shaping; introduces novel constraint classes; and enables end-to-end joint optimization of multiple heterogeneous constraints. Empirical evaluation in interpretable environments demonstrates significantly improved training stability under multi-constraint settings and yields a plug-and-play policy constraint toolkit.
📝 Abstract
Model-free reinforcement learning methods lack an inherent mechanism to impose behavioural constraints on the trained policies. Although certain extensions exist, they remain limited to specific types of constraints, such as value constraints with additional reward signals or visitation density constraints. In this work we unify these existing techniques and bridge the gap with classical optimization and control theory, using a generic primal-dual framework for value-based and actor-critic reinforcement learning methods. The obtained dual formulations turn out to be especially useful for imposing additional constraints on the learned policy, as an intrinsic relationship between such dual constraints (or regularization terms) and reward modifications in the primal is revealed. Furthermore, using this framework, we are able to introduce some novel types of constraints, allowing to impose bounds on the policy's action density or on costs associated with transitions between consecutive states and actions. From the adjusted primal-dual optimization problems, a practical algorithm is derived that supports various combinations of policy constraints that are automatically handled throughout training using trainable reward modifications. The proposed $ exttt{DualCRL}$ method is examined in more detail and evaluated under different (combinations of) constraints on two interpretable environments. The results highlight the efficacy of the method, which ultimately provides the designer of such systems with a versatile toolbox of possible policy constraints.