Problem
Research questions and friction points this paper is trying to address.
Large Language Models
Model Merging
Merging Coefficients Optimization
Evaluation Budget
SLIM: Simplex-Lattice Interpolation Merging
🤖 AI Summary
This study addresses the prohibitive computational cost of optimizing large language model merging coefficients, which typically relies on expensive benchmark evaluations. We propose a quadratic surrogate modeling approach grounded in mixture design methodology. Specifically, classical mixture designs are employed over the coefficient simplex to determine a minimal set of measurement points, replacing random sampling with structured evaluation. A quadratic surrogate model is then constructed and interpolated over the simplex to predict multi-expert mixture performance while minimizing evaluation overhead. This method accurately predicts outcomes for unseen coefficient combinations, achieving competitive model merging performance under limited computational budgets and significantly reducing the overall cost of merging optimization.
📝 Abstract
Optimizing merging coefficients for large language models can require many costly benchmark evaluations. We propose \textbf{Simplex-Lattice Interpolation Merging (SLIM)}, which constructs a quadratic surrogate of aggregate performance on the coefficient simplex using a classical mixture design. Evaluations of individual experts and equal-weight pairs determine the surrogate with the minimum number of measurements needed to identify a general quadratic on this domain. SLIM then optimizes the surrogate without further target-metric evaluations. Experiments on two model architectures demonstrate accurate prediction of unseen multi-expert mixtures and competitive merge performance under limited evaluation budgets. Matched-budget comparisons show that structured evaluation points improve prediction fidelity over random designs, including those using regularized fitting.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Simplex-Lattice Interpolation Merging
Quadratic Surrogate Model
Mixture Design
Model Merging
Evaluation Budget Optimization
🔎 Similar Papers
