No Task Vector Is an Island: A Comprehensive Study on the Composability of Task Vectors from On-Policy Distillation

2026-09-30

AI Summary

This study investigates the composability of task vectors derived from Online Policy Distillation (OPD) and Reinforcement Learning (RL) within model merging. By integrating OPD, RL, and model merging techniques, the authors analyze energy distributions in parameter space to reveal non-collinear characteristics among task vectors. The work demonstrates that OPD and RL task vectors exhibit directional complementarity, wherein even weak student-model vectors can enhance strong teacher-model performance. Crucially, it establishes that inferior standalone performance does not imply poor combinability, thereby positioning composability as an independent evaluation dimension. Cross-domain experiments further indicate that OPD-based vector composition outperforms its RL counterpart, with merged models surpassing individual components.

Abstract

Task vectors provide a simple mechanism for composing learned capabilities through model merging. However, the composability of task vectors produced by on-policy distillation (OPD) remains largely unexplored. OPD trains a student using teacher feedback on student-generated trajectories, yielding parameter updates that differ from those produced by the teacher model, usually by reinforcement learning (RL). We therefore ask whether OPD task vectors can complement their RL teacher updates and compose effectively across tasks. Across five domains and two model architectures, we find evidence for both forms of composability. Within a task, merging OPD and RL task vectors can outperform both constituent models, even when the OPD student is weaker than its RL teacher. Across tasks, OPD task-vector compositions achieve higher average scores than corresponding RL compositions in seven of eight backbone-merging-rule comparisons. Parameter-space analyses reveal substantial non-collinearity between OPD and RL updates. Experiment in CODE domain on SMOLLM3-3B shows that the combined direction outperforms either constituent direction at the tested global update norm, supporting directional complementarity in this configuration. Across tasks, OPD updates also show lower overlap among the top-10% feed-forward channels ranked by update energy. Together, these results show that weaker standalone performance does not imply weaker task-vector composability. OPD task vectors can complement stronger RL teacher updates and combine effectively across tasks, highlighting composability as a distinct property for understanding and evaluating post-training updates.

Problem

• Task Vectors
• On-Policy Distillation
• Composability
• Model Merging
• Reinforcement Learning

Innovation

• Task Vectors
• On-Policy Distillation
• Model Merging
• Composability
• Reinforcement Learning