SERPO: Self-Evolving Rubric Policy Optimization for Open-Ended Test-Time Reinforcement Learning

2026-07-29

AI Summary

This work addresses the challenge of unreliable reward signals in test-time reinforcement learning for open-ended generation tasks by proposing the first self-evolving, fully unsupervised framework. Departing from conventional answer voting mechanisms, the method establishes a tripartite co-evolutionary loop among response evidence, query-specific scoring rules, and policy parameters. It leverages a Good-Normal-Bad response archive, dynamically evolves discriminative scoring rules, and employs a likelihood-based probabilistic criterion to generate intrinsic rewards, enabling continuous policy refinement and cross-task transfer. The approach achieves performance gains of 20.63 and 20.31 points on HealthBench and ResearchQA, respectively, with an average improvement of 8.06 points across six benchmarks, while demonstrating strong out-of-domain generalization and sustained evolutionary capability.

Abstract

Test-time reinforcement learning (TTRL) enables language models to self-evolve at inference time without labeled feedback. Existing methods rely on answer voting and therefore do not extend naturally to open-ended generation, where valid responses cannot be mapped to a shared canonical answer. Without external reward models or stronger judges, adaptation must instead construct reliable rewards from the model's own outputs. We introduce SERPO (Self-Evolving Rubric Policy Optimization), which replaces answer voting with a closed loop that co-evolves response evidence, query-specific rubrics, and policy parameters. Good-Normal-Bad (G-N-B) response evolution organizes maximally separated rollouts into ordered archives; rubric evolution retains criteria that discriminate these archives; probabilistic criterion scoring converts verdict-token likelihoods into reward signals; and policy evolution optimizes the actor with the resulting signals. New actor rollouts then refresh both the archives and rubrics, closing the three-way evolution loop. Across two model configurations, two in-domain benchmarks, and four OOD benchmarks, SERPO improves HealthBench and ResearchQA by up to 20.63 and 20.31 points over the corresponding base models, raises the six-benchmark macro-average by up to 8.06 points, and supports OOD transfer and continued cross-benchmark evolution.

Problem

• test-time reinforcement learning
• open-ended generation
• self-evolution
• reward construction
• language models

Innovation

• Test-time Reinforcement Learning
• Self-Evolving Rubrics
• Open-Ended Generation
• Policy Optimization
• Reward-Free Learning