The Missing Coefficients: Bayesian Pairwise Merging for Model Personalization

2026-09-30

AI Summary

This study addresses the challenge of efficiently inferring reward weights and achieving model personalization under limited user feedback. To this end, it proposes a Bayesian pairwise merging framework that, for the first time, treats unknown reward weights as latent variables. By leveraging pairwise preferences to infer the posterior mean of these weights as model merging coefficients, the approach enables training-free, user-level personalization while quantifying coefficient uncertainty under sparse feedback. Empirical evaluations demonstrate that the proposed method significantly outperforms uniform merging baselines on tasks such as medical summarization, achieves strong alignment with simulated personality preferences, and yields confidence interval coverage rates close to nominal levels. Collectively, this work establishes a novel paradigm for preference-driven, parameter-efficient personalization.

Abstract

How can we personalize a shared expert library from a user's pairwise choices? Prior work can realize different reward trade-offs by merging reward-specialized experts, given a vector of trade-off weights. In practice, users can more naturally choose between outputs than specify numerical weights. The challenge is therefore to turn these choices into the coefficients required for merging, while accounting for ambiguity when feedback is limited. Our key idea is to treat the unknown reward weights as latent variables: infer a posterior over them from pairwise choices and reward-score differences, and use its mean directly as the merge coefficients. We instantiate this idea as Bayesian Pairwise Merging (BPM), whose posterior also characterizes which reward trade-offs remain plausible given the feedback. We evaluate BPM on radiology summarization, image captioning, and story generation, spanning text-to-text and image-to-text generation. With 100 feedback per simulated persona, BPM achieves macro decided win rates of 91.7%, 77.1%, and 64.3% against uniform merge. For six pairs of simulated personas, each prefers the model fitted to its own feedback, a pattern also observed in a human proof-of-concept. In simulations under BPM's model and prior, its nominal 90% intervals for temperature-scaled reward weights achieve task-averaged marginal coverage of 88.9% and 89.2% with only 10 and 25 comparisons, respectively. BPM thus enables personalization from pairwise feedback without per-user policy training, while characterizing the coefficient ambiguity left by limited feedback.

Problem

• Model Personalization
• Pairwise Feedback
• Expert Merging
• Reward Trade-offs
• Coefficient Ambiguity

Innovation

• Bayesian Pairwise Merging
• Model Personalization
• Pairwise Feedback
• Expert Merging
• Posterior Inference