🤖 AI Summary
This study presents the first systematic and fair comparison between model merging and joint multi-task reinforcement learning, investigating whether the former can effectively substitute for the latter. Leveraging the AppWorld benchmark, the authors train Qwen3-8B expert models of varying difficulty using the LOOP framework and apply merging techniques such as TIES and RAM+. Through analyses involving task vector cosine similarity, support set overlap, and upper/lower-bound calibration, they find that merged models achieve task completion rates comparable to jointly trained models, with no significant performance differences across merging methods. Further geometric analysis reveals that this parity stems from task vectors exhibiting substantial parameter support overlap (~65%) yet near-orthogonal directions, as evidenced by low cosine similarities ranging from 0.06 to 0.10.
📝 Abstract
Model merging is promoted as a substitute for joint multi-task training, yet in the reinforcement-learning setting this substitution is essentially never tested against the baseline it claims to replace: methods merge independently released agents precisely because a joint model is unavailable. We build the missing comparison. Training difficulty-1 and difficulty-2 Qwen3-8B specialists on the AppWorld agent benchmark with LOOP, we merge them (TIES, RAM+) and pit the result against a jointly trained model on the same data. On task-goal completion, merging matches joint RL -- and every merge variant is statistically indistinguishable. To explain why merge method does not matter here, we measure the geometry of the specialists' task vectors, which carries no task-sampling noise: they are near-orthogonal (cosine 0.06 - 0.10) despite ~65% support overlap, a small, shared direction that grows over training and that we calibrate against a random-init floor and a same-run ceiling to confirm it reflects learning, not the low-rank parameterization. Because direction and support are decoupled, support and sign-based merging (RAM, TIES) collapse to near-uniform averaging. We release all code and statistics.