Less Data, Better Alignment: Data-Centric Multi-Evaluator Agreement for Preference Optimization

2026-07-27

AI Summary

This work addresses the limitations of existing preference optimization methods, which rely heavily on large static datasets while overlooking the potential of high-quality few-shot examples. To overcome this, the authors propose DMAPO, a framework that generates candidate responses from a target policy and employs specialized multi-dimensional evaluators—assessing helpfulness, factuality, and conciseness—combined with a process-based critique mechanism to select high-consensus samples for preference training. Remarkably, DMAPO achieves strong alignment performance using only 3.45% of high-quality data, attaining a 7.50 score on MT-Bench, a 95.5% win rate in length control, and 57.3% accuracy on IFEval. It consistently outperforms baseline approaches such as SimPO across multiple benchmarks.

Abstract

Research on preference optimization often varies the training objective while holding the data fixed. We instead ask whether a small, high-confidence set of on-policy responses can provide a reliable learning signal. Our method, DMAPO (Data-centric Multi-evaluator Agreement for Preference Optimization), generates candidate responses from the target policy, evaluates helpfulness, factuality, and conciseness with rubric-specialized evaluators, applies a process-critic correction, and retains only high-consensus desirable or undesirable examples. This procedure accepts 1,871 of 54,236 Mistral-7B candidates (3.45%). KTO trained on this set reaches 7.50 on MT-Bench, 95.5% length-controlled win rate against a text-davinci-003 reference, and 57.3% IFEval prompt accuracy. Independent pairwise evaluation also favors DMAPO over SimPO: GPT-4o yields a net win rate of 23.3 points on 129 held-out prompts and 24.0 points on 200 out-of-distribution LMSYS-Chat prompts; Claude Opus 4.7 yields 24.1 points on the held-out set. Changing the evaluator model or rubric alters the selected examples but has little effect on downstream performance. A second-backbone study yields a similar 3.41% acceptance rate, although its performance gains are more modest. Across these experiments, consensus filtering offers a data-efficient route to preference optimization for general instructions, at the cost of additional curation compute and dependence on evaluator judgments.

Problem

• preference optimization
• data efficiency
• model alignment
• high-confidence data
• evaluator agreement

Innovation

• data-centric
• multi-evaluator agreement
• preference optimization
• consensus filtering
• on-policy data curation