RIMS: Preference Optimization via Smoothed Multi-pair Aggregation for Small-Scale LLM Retrieval-Augmented Generation

2026-07-17

AI Summary

This work addresses the vulnerability of small language models to noisy evidence in retrieval-augmented generation and the limitations of existing preference optimization methods, which suffer from low data efficiency or loss of gradient signals. To overcome these challenges, the authors propose RIMS, a three-stage framework: first, it generates synthetic chain-of-thought preference data for the target model via rejection sampling; second, it introduces a differentiable soft aggregation operator that replaces hard selection, preserving margin-aware discriminative structure while leveraging gradient signals from all preference pairs, with theoretical guarantees on controllable error bounds and improved gradient alignment; finally, it applies a multi-alignment algorithm for smoothed preference optimization. Experiments demonstrate that RIMS significantly improves Exact Match and F1 scores across multiple small models on four multi-hop question answering benchmarks, with particularly strong performance under noisy retrieval conditions.

Abstract

Small-scale language models (SLMs) are attractive for retrieval-augmented generation (RAG) in resource-constrained settings, but their limited capacity makes them highly sensitive to noisy or spurious retrieved evidence. Existing preference-based methods such as RoseRAG select only the hardest single preference pair via hard argmin/argmax, discarding the remaining signal; others treat multiple pairs as independent binary comparisons, resulting in low data utilization. We propose RIMS, a three-stage preference optimization framework comprising (1) synthetic chain-of-thought preference data generation via rejection sampling using the target SLM itself without relying on proprietary models, (2) a differentiable soft aggregation mechanism that replaces hard selection with a smooth operator, preserving gradient signal from all preference pairs while retaining the discriminative structure of margin-aware selection, and (3) preference optimization with the smoothed objective applied to multiple alignment algorithms. We theoretically show that the smoothed approximation admits a controllable error bound and that smooth aggregation yields provably tighter gradient alignment to the oracle objective than hard selection. Experiments on four multi-hop question answering benchmarks show that our approach outperforms state-of-the-art baselines across multiple SLM backbones, achieving consistent gains in Exact Match and F1 under noisy retrieval conditions. Our implementation is available at https://github.com/tptrix29/RIMS.

Problem

• small-scale language models
• retrieval-augmented generation
• preference optimization
• noisy retrieval
• data utilization

Innovation

• preference optimization
• smooth aggregation
• retrieval-augmented generation
• small-scale language models
• synthetic preference data