SeaPO: Strategic Error Amplification for Robust Preference Optimization of Large Language Models

2025-09-29

AI Summary

Current LLM preference optimization methods suffer from limited effectiveness due to low discriminability between positive and negative samples—especially when model scoring or generation capabilities are constrained. To address this, we propose a “Strategic Error Amplification” mechanism: it systematically identifies three canonical error patterns and controllably injects corresponding negative samples to significantly widen semantic and quality margins between positive and negative instances. Crucially, it explicitly models the error-type distribution as a guiding signal for preference learning—an innovation not previously explored. Our method integrates generative data augmentation with contrastive training, enabling multi-dimensional alignment optimization across 1.5B–14B parameter models. Experiments demonstrate consistent improvements across five core capability dimensions, with factual accuracy rising by 5–10 percentage points. Hybrid error injection further yields broad performance gains, balancing task-specificity and generalization.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Search and Optimization: Learning to Search
• Humans and AI: Learning Human Values and Preferences

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Existing alignment methods for preference optimization of large language models (LLMs) aim to enhance model performance by utilizing pairs of positive and negative samples. However, due to the limited capacity of models in scoring or generating responses, the quality of positive and negative samples may become similar during training, which complicates optimization for preference learning. To address this issue, we introduce SeaPO, a Strategic Error Amplification method that leverages three error types commonly occurring in LLMs to introduce specific error patterns into the model Preference Optimization. This strategy ensures that negative samples are more erroneous than positive samples and preference-based training is employed to mitigate the occurrence of these errors, thereby enhancing model performance. Evaluations across five capability dimensions and different model scales (1.5B to 14B) demonstrate that the generated data significantly improved overall model performance, particularly in terms of truthfulness, with improvements of 5-10 percentage points observed. Further analysis reveals that task performance varies depending on the error types introduced. Injecting the most common error types improves performance in related tasks, while a mix of error types leads to a broader performance enhancement: most tasks show stable improvements, while a few tasks exhibit significant gains.

Problem

• Addresses preference optimization challenges with similar quality samples
• Amplifies strategic error patterns to distinguish negative from positive samples
• Enhances model performance across multiple dimensions and scales

Innovation

• Amplifies strategic errors in preference optimization
• Uses three common LLM error types for training
• Enhances model truthfulness by 5-10 percentage points