CTD4 - A Deep Continuous Distributional Actor-Critic Agent with a Kalman Fusion of Multiple Critics

2024-05-04

arXiv.org

AI Summary

Traditional categorical distributional reinforcement learning (CDRL) faces challenges in continuous action spaces—including complex projection operations, difficult hyperparameter tuning, and reliance on domain knowledge. To address these, this paper proposes the first projection-free, end-to-end trainable continuous distributional model-free RL algorithm. Methodologically, it introduces an Actor-Critic architecture that directly outputs continuous probability distributions—eliminating discretization and projection—and incorporates a Kalman filter-based dynamic ensemble of multiple critics to mitigate Q-value overestimation and enhance robustness. Key contributions are: (1) the first projection-free distributional RL framework for continuous action spaces; and (2) a data-driven Kalman information fusion strategy for multi-critic ensembles. Experiments on challenging continuous control benchmarks demonstrate significantly improved sample efficiency, stable convergence, and consistent superiority over state-of-the-art distributional and non-distributional baselines.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Sequential Decision Making
• Planning, Routing, and Scheduling: Mixed Discrete/Continuous Planning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Categorical Distributional Reinforcement Learning (CDRL) has demonstrated superior sample efficiency in learning complex tasks compared to conventional Reinforcement Learning (RL) approaches. However, the practical application of CDRL is encumbered by challenging projection steps, detailed parameter tuning, and domain knowledge. This paper addresses these challenges by introducing a pioneering Continuous Distributional Model-Free RL algorithm tailored for continuous action spaces. The proposed algorithm simplifies the implementation of distributional RL, adopting an actor-critic architecture wherein the critic outputs a continuous probability distribution. Additionally, we propose an ensemble of multiple critics fused through a Kalman fusion mechanism to mitigate overestimation bias. Through a series of experiments, we validate that our proposed method provides a sample-efficient solution for executing complex continuous-control tasks.

Problem

• Simplifies distributional RL implementation
• Addresses overestimation bias with Kalman fusion
• Tailors for continuous action spaces efficiently

Innovation

• Continuous Distributional Model-Free RL algorithm
• Actor-critic architecture with continuous probability distribution
• Kalman fusion mechanism for multiple critics