QLPO: Quadrant-weighted Sampling for Length-aware Policy Optimization

2026-07-23

AI Summary

This work addresses the challenge of excessive reasoning chain length in large reasoning models within reinforcement learning, which incurs high latency and deployment costs. Existing length-control approaches often rely on explicit penalties or auxiliary modules, requiring careful hyperparameter tuning and risking degradation of reasoning quality. To overcome these limitations, the authors propose QLPO, a method that over-generates candidate responses and employs a quadrant-weighted sampling mechanism to implicitly favor shorter correct and longer incorrect samples while preserving the balance between correct and incorrect experiences. Without modifying the reward function, QLPO integrates GRPO-based resampling with policy optimization, achieving 30%–70% response length compression across models ranging from 1.5B to 32B parameters, while maintaining or even improving reasoning accuracy—thereby substantially enhancing the trade-off between accuracy and response length.

Abstract

Recent large reasoning models often develop long chain-of-thought responses during reinforcement learning (RL), resulting in high inference latency and deployment cost. Existing methods for response length control typically rely on explicit length penalties or additional control modules, which require careful tuning and may compromise reasoning quality. We propose Quadrant-weighted Sampling for Length-aware Policy Optimization (QLPO), a simple resampling-based variant of GRPO that introduces implicit length control without modifying the reward function. QLPO first over-generates candidate responses and then resamples the training group by preserving the empirical correct/incorrect ratio while favoring short correct responses and long incorrect responses. This reshapes the training distribution and implicitly encourages shorter model outputs. Across models ranging from 1.5B to 32B parameters, including both base models and strong reasoning models, QLPO consistently improves the accuracy-length trade-off. It reduces response length by 30% to 70% while preserving reasoning performance. These results suggest that structured resampling provides an effective and robust approach to efficient reasoning.

Problem

• response length control
• reasoning efficiency
• inference latency
• length-aware optimization
• chain-of-thought

Innovation

• length-aware policy optimization
• quadrant-weighted sampling
• response length control
• resampling-based RL
• efficient reasoning