CT-Merging: Consensus Directions and Task-Level Scaling for LoRA Adapter Merging

2026-07-18

AI Summary

Deploying multi-task LoRA adapters typically requires storing them separately and switching between them during inference, incurring substantial memory and computational overhead. To address this challenge, this work proposes an efficient fusion method that constructs a consensus direction via subspace projection averaging and decouples direction selection from task-specific RMS coefficient scaling, thereby reducing reliance on the ordering of singular values from SVD. Evaluated on the DC-Merge CLIP benchmark, the proposed approach significantly improves performance, achieving average normalized accuracy gains of 2.56 and 1.51 percentage points for ViT-B/32 and ViT-L/14 models, respectively.

Abstract

LoRA adapters provide an efficient way to specialize a pretrained model for many downstream tasks, but deploying one adapter per task requires adapter storage and task selection at inference time. Model merging addresses this issue by combining independently trained adapters into one multi-task adapter. Recent SVD-based LoRA merging methods mainly focus on constructing shared or task specific directions, while the coefficients assigned to the final directions are often directly from the original task SVD. On a fixed merged basis, inherited coefficients preserve component order with high rank correlation, yet their magnitudes differ substantially from the coefficients induced by the task updates. To address this mismatch, we propose CT-Merging, a LoRA-aware merging algorithm that estimates consensus directions from average task subspace projectors and assigns task-level RMS coefficient scales in the final update. CT-Merging uses repeated support across task SVD subspaces to construct the common basis, while reducing reliance on rank wise SVD magnitudes after direction construction. On the DC-Merge CLIP adapter benchmark, CT-Merging achieves superior average normalized accuracy compared to state-of-the-art merging methods and further improves over DC-Merge by 2.56 points on ViT-B/32 and 1.51 points on ViT-L/14 KnoTS-trained checkpoints.

Problem

• LoRA adapter merging
• multi-task learning
• model merging
• adapter compression
• parameter-efficient fine-tuning

Innovation

• LoRA merging
• consensus directions
• task-level scaling
• adapter fusion
• subspace projection