Exposure-Based Reinforcement Learning to Rank

2026-07-21

AI Summary

This work addresses the inefficiency of traditional reinforcement learning in ranking tasks, which stems from an excessively large action space and reliance on handcrafted gradient computations that hinder compatibility with automatic differentiation. The authors propose a novel reinforcement learning framework for ranking based on document exposure distributions, reframing policy gradient estimation as a differentiable operation over the exposure distribution and encapsulating it into an end-to-end differentiable loss function—eliminating the need for custom gradient implementations. By incorporating variance reduction, baseline correction, and partial marginalization, the method substantially improves sample efficiency and training stability. Designed for plug-and-play integration, it incurs no additional computational overhead on GPUs and demonstrates faster convergence and superior ranking performance compared to existing approaches.

Abstract

Reinforcement learning (RL) methods for learning-to-rank (LTR) can optimize (almost) any ranking goal, e.g., from precision or discounted cumulative gain to fairness-of-exposure or ranking distillation. However, standard RL is ineffective and computationally costly due to the enormous action space in LTR settings. Existing methods reach computational efficiency through custom gradient computation algorithms, but they are very complex to implement and often clash with auto-differentiation. Consequently, existing RL for LTR is not attractive to many practitioners. We reconsider RL for LTR while actively avoiding reliance on custom gradients. Contrary to the existing approaches, we focus on variance reduction and GPU computation. In doing so, we discover that high sample-efficiency can be reached through baseline corrections and partial marginalization. Furthermore, we propose an abstraction that places gradient estimation behind a document-exposure distribution, this enables seamless plug-and-play integration with auto-differentiation. Thereby, one only has to implement a loss as a differentiable function of exposure and RL for LTR can optimize it using auto-differentiation. Our experimental results reveal that our new exposure-based RL for LTR approach converges considerably faster and at significantly higher ranking performance than existing custom gradients, with no additional costs in computation time when using GPUs. In contrast, existing custom gradients result in severe stability issues when converging over many epochs, which never occur for our methods. Thus, we considerably improve RL for LTR methodology by increasing its effectiveness, efficiency, and ease of application.

Problem

• Reinforcement Learning
• Learning-to-Rank
• Exposure
• Auto-differentiation
• Ranking

Innovation

• exposure-based reinforcement learning
• learning-to-rank
• variance reduction
• auto-differentiation
• gradient estimation