From "Weak" Signals to Strong Models: Preference Delta Aggregation with LoRA Merging

2026-05-29

AI Summary

This study addresses the challenge of effectively aggregating multiple low-quality yet relatively reliable preference signals from weak models to enhance the performance of strong language models. To this end, the authors propose the Preference Delta Aggregation (PDA) framework, which extracts preference discrepancy signals between weak–weaker model pairs to train LoRA adapters. Furthermore, they introduce Geometric Alignment Merging (GAM), a novel method that mitigates directional interference during LoRA merging, thereby enabling robust integration of complementary capabilities. This work represents the first successful approach to effectively aggregate multiple weak preference signals. Experimental results demonstrate that PDA achieves average improvements of 6.8 and 7.3 points on knowledge reasoning and agent search tasks, respectively, significantly outperforming all single- and multi-signal baselines, with a maximum gain of 4.3 points over the best single-signal baseline.

Abstract

Training strong large language models (LLMs) requires high-quality supervision, which is often scarce. Recent work shows that paired preference data from weak-weaker model pairs (e.g., Qwen3 4B over 1.7B), despite the limited quality of individual responses, can provide an effective supervision signal through relative quality deltas, which we term a "weak" signal. This motivates a key research question: can multiple "weak" signals be constructively aggregated for improving strong models (e.g., Qwen3 8B)? To this end, we propose Preference Delta Aggregation (PDA), the first framework that derives a preference delta from each weak-weaker model pair, instantiates it as a LoRA adapter learned through preference optimization, and aggregates the resulting deltas via LoRA merging. To further mitigate directional interference during LoRA merging, we introduce Geometric Alignment Merging (GAM), a geometry-aware merging method that aligns adapter subspaces before aggregation, enabling more robust composition of diverse deltas. Evaluations on knowledge reasoning and agentic search benchmarks show that aggregating multiple "weak" signals pushes performance beyond any single signal, with further gains as additional signals are incorporated. Correspondingly, PDA with GAM improves the strong model by 6.8 and 7.3 points on average for knowledge reasoning and agentic search, respectively. It outperforms all single-delta and multi-delta baselines, exceeding the best single-delta baseline by 2.1 and 4.3 points. Further analysis attributes these gains to the effective composition of complementary capabilities encoded across distinct preference deltas.

Problem

• weak signals
• preference aggregation
• large language models
• model supervision
• preference delta

Innovation

• Preference Delta Aggregation
• LoRA Merging
• Geometric Alignment Merging
• Weak Supervision Signals
• Preference Optimization