S-SPPO: Semantic-Calibrated Self-Play Preference Optimization

2026-05-31

AI Summary

This work addresses the issue of policy degradation in Self-Play Preference Optimization (SPPO), which arises when semantically similar responses are overconfidently deemed superior. To mitigate this, the authors propose a dual-space semantic calibration framework that dynamically adjusts win-rate targets in preference space via semantic gating and introduces a latent repulsion mechanism in representation space to preserve response diversity. This approach is the first to integrate semantic calibration into SPPO while preserving the constant-sum game structure necessary for convergence to a Nash equilibrium. Experimental results demonstrate that, using only self-generated preference data and without any human annotations, Llama-3-8B achieves a 52.19% win rate and a 47.46% length-controlled win rate on AlpacaEval 2.0.

Abstract

Aligning Large Language Models (LLMs) with human preferences is often formulated via Direct Preference Optimization (DPO). However, the standard Bradley-Terry instantiation of DPO is limited in modeling common departures from transitivity in human preferences. To address this, recent work has introduced Self-Play Preference Optimization (SPPO), which iteratively refines the policy by training on self-generated win-lose pairs. Our investigation, however, reveals a critical instability in SPPO: the optimization is prone to policy degeneration when the preference oracle assigns overly confident wins to semantically indistinguishable responses. To mitigate this, we propose S-SPPO, a dual-space semantic calibration framework comprising: i) Supervision Calibration via semantic gating, which anneals win rate targets toward the maximum-entropy baseline as semantic overlap increases; and ii) Representation Calibration via latent repulsion to enforce geometric diversity to prevent manifold collapse and maintain latent diversity between chosen and rejected samples. Theoretically, we show that the calibration preserves the constant-sum game structure, facilitating convergence to a Nash Equilibrium. Empirically, S-SPPO avoids the performance degradation seen in prior methods, achieving 52.19% win rate and 47.46% length-controlled win rate on AlpacaEval 2.0 with Llama-3-8B, without using additional human-annotated preferences during training. The code will be available at https://github.com/xiwenc1/s-sppo.

Problem

• preference optimization
• policy degeneration
• semantic indistinguishability
• self-play
• large language models

Innovation

• Semantic Calibration
• Self-Play Preference Optimization
• Latent Repulsion
• Preference Alignment
• Nash Equilibrium