Orthogonal Yet Coupled: Decoupling Geometric Components for Model Merging

2026-09-29

AI Summary

This study addresses the limitation of existing model merging methods that treat task vectors as indivisible units, causing coupled interference among geometric components and constraining merging performance. To overcome this, we propose DiGA, a novel geometry-aware decoupled merging framework. Using pretrained weights as a reference, DiGA projects task vectors into distinct geometric subspaces via orthogonal decomposition, independently aggregates each component, and reassembles them into a unified update, thereby eliminating cross-component coupling. This approach is highly generalizable and can be seamlessly integrated into various existing merging algorithms. Extensive experiments across multiple models and tasks demonstrate that DiGA significantly enhances merged model performance while effectively mitigating capability degradation.

Abstract

Merging pretrained models has emerged as an effective approach for consolidating diverse capabilities into a single unified model. However, prevailing merging methods typically treat each task vector as an indivisible merging unit, overlooking the heterogeneous geometric changes encoded within it. This treatment can induce cross-component coupling: when merging decisions are derived from statistics of the complete task vector, the geometric characteristics of one component may influence how another is selected, weighted, or combined, potentially degrading the quality of the merged model. To address this issue, we propose DiGA, a Disentangled Geometry-Aware model merging framework. Using the pretrained weights as a shared geometric reference, DiGA orthogonally decomposes each task vector into components corresponding to distinct geometric attributes. Rather than merging the task vectors as a whole, DiGA aggregates corresponding components independently within their respective subspaces and subsequently recombines them into a unified update. This component-wise formulation preserves the geometric identity of each component and prevents the characteristics of one component from interfering with the aggregation of another. Furthermore, DiGA can be incorporated into a broad range of existing model merging methods. Extensive experiments across diverse models, tasks, and merging methods demonstrate that DiGA improves merged-model performance and reduces capability degradation. Our repository is on https://github.com/wzj1718/DiGA.

Problem

• Model Merging
• Task Vector
• Cross-component Coupling
• Geometric Decomposition
• Pretrained Models

Innovation

• Model Merging
• Task Vector Disentanglement
• Orthogonal Decomposition
• Geometry-Aware Aggregation
• Cross-component Decoupling