actor-critic methods

This work addresses the challenge of rapid adaptation to new goals in complex reinforcement learning control tasks by proposing a bilinear Actor-Critic framework grounded in shared low-dimensional goal embeddings. The policy and value function are jointly modeled as bilinear forms composed of shared basis functions and goal-specific coefficients. Inspired by neural gain modulation mechanisms, the approach enables zero-shot transfer without retraining. Evaluated on the MuJoCo Ant multi-directional locomotion task, the model generalizes effectively to unseen movement directions, achieving significant improvements in immediate adaptability for high-dimensional control through specialized policy basis heads and interpolation in the goal embedding space.

Existing Actor-Critic (AC) algorithms face a fundamental trade-off between greediness and stability during policy improvement: gradient-based updates tend to be overly conservative, whereas aggressive greedy actions often destabilize training. To address this, we propose the Value-Improved AC (VI-AC) framework—the first AC architecture to jointly and concurrently improve both policy (actor) and value (critic) functions, thereby unifying strengths of policy-based and value-based methods. We instantiate VI-AC as two plug-and-play algorithms—VI-TD3 and VI-DDPG—that integrate Bellman operator-driven value enhancement, target networks, and delayed updates for stable off-policy learning. Evaluated across all MuJoCo benchmark environments, both variants achieve significantly improved sample efficiency and training stability, matching or surpassing state-of-the-art baselines. These results empirically validate the effectiveness and generalizability of the dual-improvement paradigm in AC frameworks.

This paper addresses the instability and divergence issues in value function learning within offline reinforcement learning. We propose KROPE, a bisimulation-based kernel representation method that constrains state-action representations to jointly preserve reward similarity and next-state dynamics similarity, thereby implicitly enforcing contraction of the Bellman operator in the representation space. We provide the first theoretical proof that state-action representations satisfying the bisimulation metric guarantee convergence of offline value function learning. Building on this insight, we design the KROPE algorithm to jointly optimize representation learning and value estimation in an end-to-end manner. Experiments demonstrate that KROPE significantly reduces Bellman error and policy evaluation bias, achieving superior stability and accuracy over leading offline RL methods across multiple benchmarks.

This work addresses the problem of recovering environment transition dynamics directly from a converged value function, bypassing explicit rule learning or model construction. Methodologically, it algebraically reconstructs the Bellman equation, analyzes value function gradients, and derives identifiability conditions to establish a theoretical mapping from value functions to transition models. It provides the first theoretical proof that, under next-state identifiability—e.g., local injectivity of the value function with respect to action–state pairs—the optimal or converged value function implicitly encodes complete dynamical information; it further proposes a model-free dynamics inversion algorithm. Experiments demonstrate accurate transition model reconstruction on canonical MDPs. The contribution establishes a novel paradigm bridging model-free and model-based reinforcement learning, advancing both the theory and practice of “reading world models from value functions.”

This work addresses the challenge of zero-shot cross-task transfer in reinforcement learning. We propose the Function Encoder framework, which maps reward and state-transition functions into low-dimensional, semantically consistent task embeddings via weighted combinations of nonlinear basis functions—enabling task alignment and immediate transfer without online fine-tuning. The framework is modular and seamlessly integrates with mainstream RL algorithms including PPO, SAC, and DQN. Experiments across multiple benchmark domains demonstrate substantial improvements in zero-shot generalization, achieving state-of-the-art performance in data efficiency, asymptotic policy quality, and training stability. Our core contribution is the first explicit encoding of task-level functional representations (i.e., reward and dynamics functions) into transferable vector embeddings—departing from conventional paradigms that rely solely on policy- or value-function-based transfer. This paradigm shift enables more principled and scalable cross-task knowledge reuse.

Temporal-difference (TD) learning traditionally enforces value-function matching only at the Bellman target’s scalar values, neglecting its local geometric structure—particularly first-order derivatives with respect to state and action. Method: We propose the *first-order Bellman consistency constraint*, requiring the value function to match not only the Bellman target’s value but also its gradients in state and action. Leveraging differentiable environment dynamics, we analytically derive gradient targets and introduce a Sobolev-type loss that jointly optimizes both value and derivative terms. The method is seamlessly integrated into standard Actor-Critic frameworks (e.g., DDPG, SAC) without architectural modification. Contribution/Results: This is the first explicit incorporation of first-order Bellman consistency into TD learning. It significantly enhances the critic’s ability to capture local geometry of the target function, accelerates critic convergence, improves policy gradient stability, and maintains full compatibility with existing algorithms—demonstrating strong practical utility.

This work addresses the lack of a unified theoretical framework for reinforcement learning, which has hindered systematic analysis of its convergence, sample complexity, and generalization. Building upon Markov decision processes and Bellman operators, the paper introduces a cohesive analytical framework that integrates tools from operator theory, stochastic approximation, convex duality, and function approximation. This framework encompasses a broad range of algorithms, including value iteration, policy iteration, temporal difference methods, off-policy learning, and constrained MDPs. By leveraging contraction mappings, monotone operators, martingale techniques, mirror/proximal optimization, concentration inequalities, and mixing process theory, the study establishes finite-sample performance bounds and asymptotic convergence guarantees for diverse reinforcement learning algorithms, thereby forging a rigorous theoretical bridge between probability theory, optimization, and statistics.

This study addresses the challenges of deploying Actor-Critic algorithms in real-world control systems, where poor reliability and high sensitivity to hyperparameters often hinder practical application. Focusing on a real-world water treatment plant control task, the authors conduct over 33,000 large-scale ablation experiments to systematically evaluate how key algorithmic components—such as policy update schemes, action distribution representations, gradient estimation methods, and update frequencies—affect performance stability and hyperparameter robustness. Their empirical analysis reveals, for the first time, that commonly adopted default configurations (e.g., Gaussian action distributions with pathwise derivatives) exhibit low reliability, whereas bounded action distributions combined with adaptive update strategies substantially enhance robustness. The work identifies high-stability algorithmic configurations that significantly reduce performance variance under limited tuning budgets, offering actionable, component-level design guidelines for industrial deployment.

Traditionally, model-free and model-based reinforcement learning have been regarded as mutually exclusive paradigms, with ambiguity surrounding whether value functions implicitly encode environmental dynamics. This work proposes P-learning, a novel approach that establishes, for the first time, an inverse mapping from goal-conditioned Q-functions to the environment’s transition kernel by inverting the Bellman equation. The method enables a unified analysis across both stochastic and deterministic MDPs, as well as discrete and continuous state spaces. By revealing that agents can implicitly learn a world model from sparse reward signals alone, P-learning bridges the conceptual divide between model-free and model-based RL. Empirical validation in Reacher, MountainCar, and FourRooms environments demonstrates its ability to achieve near-optimal generalization to out-of-distribution goals.

This work proposes a relative value learning framework that shifts the focus from absolute value estimation—commonly used in traditional reinforcement learning—to directly modeling pairwise relative value differences, which are sufficient for policy optimization. The framework introduces an antisymmetric function to represent value differences between state pairs and defines a novel pairwise Bellman operator with a unique fixed point. Building upon this foundation, the authors derive n-step and λ-return objectives and develop an unbiased Relative Generalized Advantage Estimator (R-GAE) for policy gradient computation. When integrated into Proximal Policy Optimization (PPO), the approach achieves performance on par with standard PPO across 49 Atari games, demonstrating that relative value estimation can serve as an effective alternative to absolute value critics.