Categorical Policies: Multimodal Policy Learning and Exploration in Continuous Control

2025-08-19

AI Summary

Most existing deep reinforcement learning policies employ unimodal Gaussian distributions, limiting their ability to handle sparse-reward settings and complex dynamics that demand multimodal behaviors and structured exploration. To address this, we propose **Categorical Policies**, the first approach to integrate discrete categorical latent variables into continuous-control policies. Our method employs Gumbel-Softmax differentiable sampling and a conditional action-generation network to enable end-to-end multimodal policy learning. This design supports context-adaptive decision-making and robust exploration, substantially enhancing policy expressivity and training stability. Empirical evaluation on the DeepMind Control Suite demonstrates that our approach achieves faster convergence and superior asymptotic performance compared to standard Gaussian policies. These results validate the critical role of discrete categorical distributions in improving both exploration efficiency and behavioral diversity in continuous control.

Technology Category

• Machine Learning: Multimodal Learning
• Search and Optimization: Mixed Discrete/Continuous Search
• Reasoning under Uncertainty: Sequential Decision Making

Application Category

• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

A policy in deep reinforcement learning (RL), either deterministic or stochastic, is commonly parameterized as a Gaussian distribution alone, limiting the learned behavior to be unimodal. However, the nature of many practical decision-making problems favors a multimodal policy that facilitates robust exploration of the environment and thus to address learning challenges arising from sparse rewards, complex dynamics, or the need for strategic adaptation to varying contexts. This issue is exacerbated in continuous control domains where exploration usually takes place in the vicinity of the predicted optimal action, either through an additive Gaussian noise or the sampling process of a stochastic policy. In this paper, we introduce Categorical Policies to model multimodal behavior modes with an intermediate categorical distribution, and then generate output action that is conditioned on the sampled mode. We explore two sampling schemes that ensure differentiable discrete latent structure while maintaining efficient gradient-based optimization. By utilizing a latent categorical distribution to select the behavior mode, our approach naturally expresses multimodality while remaining fully differentiable via the sampling tricks. We evaluate our multimodal policy on a set of DeepMind Control Suite environments, demonstrating that through better exploration, our learned policies converge faster and outperform standard Gaussian policies. Our results indicate that the Categorical distribution serves as a powerful tool for structured exploration and multimodal behavior representation in continuous control.

Problem

• Addressing unimodal policy limitations in deep reinforcement learning
• Enabling robust exploration in continuous control environments
• Modeling multimodal behavior with categorical distribution for adaptation

Innovation

• Categorical Policies model multimodal behavior modes
• Uses latent categorical distribution for mode selection
• Ensures differentiable discrete structure for optimization