Co-Evolving LLM Evaluators and Policies via DynamicRubric

2026-07-22

AI Summary

This work addresses the challenge in large language model (LLM) policy optimization where diminishing quality differences among generated samples hinder evaluators from providing effective supervisory signals. To this end, we propose DynamicRubric, a novel framework that, for the first time, reveals—from a probabilistic allocation perspective—the critical role of evaluator score discrepancies in policy updates. It introduces a response-set-conditioned, dynamically weighted binary scoring mechanism that enables co-evolution between the evaluator and the policy. Using only an 8B-parameter model to replace conventional large static reward models, our method significantly outperforms baselines employing 70B reward models or 235B static rubrics on both reasoning and code generation tasks. The approach has been deployed in WeChat Search’s AI question-answering system, serving tens of millions of daily requests and yielding substantial improvements in key online metrics.

Abstract

Post-training with evaluator feedback on policy-induced samples serves as a major mechanism for improving large language models. As policies improve, these sampled responses become close in quality. These close candidates create a bottleneck for policy optimization: collapsed relative evaluator score gaps yield weak or misleading policy supervision. We theoretically characterize why these gaps matter through a probability allocation view, showing that the directional gain of shifting probability mass from one response to another is exactly the evaluator score gap between them. This identifies relative score gaps as the policy optimization signals that guide updates. Motivated by this view, we propose DynamicRubric, a response-set-conditioned evaluator--policy co-evolution framework that generates weighted binary rubric items for each candidate set and aggregates the resulting judgments into response-level scores. In our experiments with 8B backbones, DynamicRubric improves evaluator performance and provides stronger policy supervision than baselines using a 70B reward model or a 235B static rubric generator. DynamicRubric-optimized policies also show gains on verifiable reasoning and coding tasks. A DynamicRubric-optimized model is fully deployed in WeChat Search's AI answering scenario, where it serves all online traffic across tens of millions of requests per day and improves key online metrics. These results suggest a principle for evaluator-guided post-training: evaluators should evolve with the policies they supervise.

Problem

• policy optimization
• evaluator feedback
• score gap collapse
• post-training
• LLM evaluation

Innovation

• DynamicRubric
• evaluator-policy co-evolution
• relative score gaps
• response-set-conditioned evaluation
• post-training with feedback