Support Vector Rubrics: Closing the Gap Between Self-Generated and Human Rubrics

2026-06-06

AI Summary

This work addresses the limited discriminative power of existing automatic rubric generation methods on challenging examples, which struggle to differentiate between outputs from similar language models. The authors propose a novel rubric construction paradigm centered on maximizing decision margins, introducing support vector concepts into large language model evaluation for the first time. By leveraging contrastive feature mining, they build a rubric library and iteratively refine it using a prompt-conditioned selector and global weighting scheme. Integrating adversarial hard-negative probing with preference data learning, their method reduces the gap between automatically generated and human-defined rubrics on RubricBench from 24.1 to just 0.3 points, significantly outperforming current automatic scoring and evaluation baselines. Notably, the resulting rubric library enables cross-rater transferability without retraining and demonstrates strong performance across multiple reward modeling benchmarks.

Abstract

Rubric-based evaluation is a promising paradigm for judging large language model (LLM) outputs, yet self-generated rubrics lag human-annotated criteria on hard instances. We argue this discriminative gap reflects an objective mismatch: self-generated rubrics describe good responses, whereas effective criteria must discriminate between close candidates. To close this gap, we introduce SVR (Support Vector Rubrics), a framework that recasts rubric construction as max-margin boundary learning over preference data. SVR mines contrastive features from preference pairs into a rubric bank, learns a prompt-conditioned selector together with global rubric weights, and iteratively refines the bank through support-pair selection and adversarial probing of hard negatives. At inference, given only the prompt, SVR retrieves the top-rubrics from the bank and scores responses. On RubricBench, SVR narrows the gap to human reference rubrics from 24.1 to 0.3 points and outperforms strong self-rubric and judge baselines, and the learned bank transfers across judges without retraining. On RewardBench 1&2, and RM-Bench, it remains competitive with dedicated reward models, demonstrating broader reward modeling capability. Overall, boundary-defining rubrics offer a principled route to closing the discriminative gap in LLM evaluation.

Problem

• rubric-based evaluation
• self-generated rubrics
• discriminative gap
• large language models
• preference data

Innovation

• Support Vector Rubrics
• max-margin boundary learning
• contrastive feature mining
• rubric bank
• adversarial probing