Policy gradient methods for ordinal policies

2025-06-23

AI Summary

Standard softmax policies in reinforcement learning fail to model inherent ordinal relationships among discrete actions. To address this, we propose a novel policy parameterization grounded in ordinal regression—the first such integration of ordinal regression into RL policy design. Our method explicitly encodes the ordinal structure of the action space, overcoming the limitation of conventional approaches that ignore sequential action dependencies. Embedded within the policy gradient framework, it naturally accommodates discretized continuous-action tasks. Empirical evaluation across multiple industrial scenarios and standard continuous-control benchmarks demonstrates substantial improvements in sample efficiency and policy performance; notably, it remains highly competitive even after action discretization. The core contribution lies in establishing a unified modeling paradigm that jointly incorporates ordinal constraints and policy learning, thereby bridging structured prediction with reinforcement learning.

Technology Category

• Machine Learning: Reinforcement Learning
• Search and Optimization: Learning to Search
• Intelligent Robots: Learning & Optimization for ROB

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

In reinforcement learning, the softmax parametrization is the standard approach for policies over discrete action spaces. However, it fails to capture the order relationship between actions. Motivated by a real-world industrial problem, we propose a novel policy parametrization based on ordinal regression models adapted to the reinforcement learning setting. Our approach addresses practical challenges, and numerical experiments demonstrate its effectiveness in real applications and in continuous action tasks, where discretizing the action space and applying the ordinal policy yields competitive performance.

Problem

• Softmax fails to capture action order in RL
• Proposes ordinal regression for RL policy parametrization
• Solves real-world industrial and continuous action tasks

Innovation

• Ordinal regression models for policy parametrization
• Adapts to reinforcement learning setting
• Effective in real and continuous action tasks