Enhancing Rubric-based RL via Self-Distillation

2026-07-20

AI Summary

This work addresses the challenges in rubric-based reinforcement learning where unexplored criteria (UC) lack optimization signals and suppressed criteria (SC) suffer from diminished learning signals. To tackle these issues, the authors propose Criterion-aware Policy Optimization (CriPO), a novel approach leveraging an internal self-distillation mechanism without external supervision. CriPO employs a criterion-injected self-teacher for UC and a counterfactual self-teacher for SC, and it is the first to systematically identify and quantify the SC phenomenon. The method integrates on-policy self-distillation, a local forward KL divergence loss, and counterfactual advantage reweighting to enable fine-grained, criterion-level optimization while mitigating train–test inconsistency. Experiments demonstrate that CriPO significantly outperforms existing methods on medical and scientific benchmarks, achieving superior performance with approximately 50% fewer optimization steps.

Abstract

Rubric-based RL has recently shown promise in improving LLMs on open-ended tasks. A widely recognized limitation of rubric-based RL is limited exploration: criteria that no rollout manages to satisfy (Unexplored Criteria, UC) receive no optimization signal. Recent methods address this by incorporating rubric information as external guidance during rollout, yet they introduce a train-inference mismatch: the policy is optimized on rollouts produced under external guidance while this guidance is absent at inference time, causing error accumulation through autoregressive decoding. Moreover, these exploration-focused approaches overlook a fundamentally different failure mode that we term Suppressed Criteria (SC) -- criteria that are satisfied by some rollouts yet whose learning signals are lost during optimization because scalar reward aggregation assigns them non-positive aggregate advantages. Our analysis reveals that SC are remarkably prevalent: over 57% of samples exhibit this failure mode throughout training, with an average of 1.8 SC per sample. To simultaneously address both UC and SC without introducing training-inference mismatch, we propose Criterion-Distilled Policy Optimization (CriPO), which enhances rubric-based RL via on-policy self-distillation. For UC, CriPO constructs a criterion-injection self-teacher and computes a localized forward-KL loss to inject missing behaviors into the policy. For SC, CriPO employs a counterfactual self-teacher to locate criterion-relevant tokens in negative-advantage rollouts and flips their token-level advantages to positive values, preserving useful patterns that would otherwise be suppressed. Experiments on medicine and science benchmarks demonstrate that CriPO consistently outperforms rubric-based RL, achieving stronger final performance with approximately $2\times$ fewer optimization steps.

Problem

• rubric-based RL
• Unexplored Criteria
• Suppressed Criteria
• train-inference mismatch
• reward aggregation

Innovation

• rubric-based RL
• self-distillation
• unexplored criteria
• suppressed criteria
• policy optimization