RubricArmor: Adversarial Evolution Improves LLM-Based Rubric Generation

2026-10-04

AI Summary

This study addresses the vulnerability of large language model (LLM)-generated rubrics to reward hacking, which enables low-quality responses to receive undeservedly high scores. To mitigate this issue, we propose RubricArmor, an adversarial framework that pioneers the integration of an adversarial evolution mechanism during the rubric generation phase. By combining reinforcement learning with adversarial prompt engineering, the framework employs an iterative attack-simulation and repair-evolution algorithm to proactively identify and patch potential vulnerabilities, thereby achieving active defense at the generation stage rather than merely optimizing rubric granularity. Experimental results demonstrate that our approach significantly outperforms existing baselines and effectively enhances downstream rubric-based RLHF alignment performance.

Abstract

Rubric-based reinforcement learning (RL) provides interpretable rewards for aligning large language models (LLMs) by evaluating responses against query-specific evaluation criteria. To construct rubrics at scale, a straightforward approach to LLM-based rubric generation is to prompt an LLM to generate a rubric directly from the query. However, rubrics directly generated by LLMs are vulnerable to reward hacking, since omitted or underspecified criteria allow the policy to obtain high rubric rewards with low-quality responses. Existing LLM-based rubric generation methods improve the granularity and coverage of the generated criteria but do not proactively guard against reward hacking. To address this limitation, we propose RubricArmor, an adversarial framework that exposes and mitigates potential reward hacking at the rubric generation stage before it occurs in subsequent RL. Specifically, RubricArmor performs adversarial evolution, in which an attack step and a repair step alternate over multiple rounds. The attack step simulates the reward hacking of the policy by constructing adversarial responses that satisfy the current rubric but fail to properly complete the task. The repair step then revises the rubric to detect the response defects exposed by the attack step while preserving other valid criteria. Extensive experiments demonstrate that RubricArmor outperforms competitive rubric generation baselines and translates into more effective downstream rubric-based RL.

Problem

• Rubric-based reinforcement learning
• Reward hacking
• LLM-based rubric generation
• Large language models

Innovation

• Adversarial Evolution
• Rubric Generation
• Reward Hacking
• Reinforcement Learning
• Large Language Models