Structure-aware Relative Policy Optimization for Ranking

2026-07-28

AI Summary

Existing reinforcement learning–based ranking methods treat permutations as atomic actions and rely solely on scalar rewards, thereby ignoring the structural relationships among permutations. This oversight leads to inaccurate credit assignment and unstable policy updates. To address this limitation, this work proposes the Structured Ranking Policy Optimization (SRPO) framework, which explicitly models structural differences between permutations for the first time in ranking policy optimization. SRPO introduces a top-weighted Kendall-tau distance to quantify permutation dissimilarity and uses it to normalize pairwise reward differences, enabling precise policy updates guided by unit structural changes. By emphasizing local optimization at top-ranked positions, the method significantly improves both performance and training stability in listwise ranking tasks under limited feedback and complex evaluation settings.

Abstract

Ranking is a fundamental component of modern information access systems. Reinforcement learning (RL) provides a flexible framework for directly optimizing coarse-grained feedback and system-level objectives defined over the complete ranking list. However, existing RL-based ranking methods typically treat each sampled permutation as an atomic output and evaluate it primarily through a scalar reward, overlooking the structural relationships among different ranking lists. Consequently, permutations with similar rewards but substantially different permutation patterns may receive comparable optimization signals, potentially leading to inaccurate credit assignment and overly aggressive policy updates. To address this limitation, we propose SRPO, a \textbf{S}tructure-aware \textbf{R}elative \textbf{P}olicy \textbf{O}ptimization framework for listwise ranking. SRPO measures the discrepancy between sampled permutations using a top-weighted Kendall-tau distance and normalizes their pairwise reward differences by the corresponding distances. It quantifies the reward improvement per unit of ranking change, thereby emphasizing efficient local refinements, particularly those involving top-ranked positions. Experimental results across two ranking scenarios demonstrate that explicitly modeling permutation-level differences improves the effectiveness and stability of listwise ranking, with particularly favorable performance in limited-feedback and complex list-level optimization settings.

Problem

• ranking
• reinforcement learning
• permutation structure
• credit assignment
• listwise optimization

Innovation

• Structure-aware
• Relative Policy Optimization
• Listwise Ranking
• Kendall-tau distance
• Reinforcement Learning