CD-LoRA: Consistency-Driven Low-Rank Adaptation for Multi-Task Fine-Tuning

2026-08-22

AI Summary

本文针对多任务微调中基于LoRA方法的训练-推理不一致问题，提出了一种无路由机制的Consistency-Driven Low-Rank Adaptation（CD-LoRA）方法，通过一致性驱动的对齐机制提升模型稳定性。

Abstract

While Multi-Task Learning (MTL) is essential for adapting Large Language Models (LLMs) to diverse domains, prevailing LoRA-based methods rely on complex routing mechanisms that partition task-specific knowledge. In this work, we reveal that such routing-based designs are prone to a training-inference discrepancy, where stochastic routing decisions under distribution shifts compromise inference stability. Driven by a second-order Taylor analysis that exposes the instability induced by routing variance, we challenge the training-inference discrepancy and propose Consistency-Driven Low-Rank Adaptation (CD-LoRA). By eliminating routers entirely, CD-LoRA employs a consistency-driven alignment mechanism to enforce representation congruence across tasks in a shared low-rank space. This paradigm fosters robust, task-agnostic features without explicit partitioning overhead. Extensive experiments show that CD-LoRA consistently outperforms state-of-the-art multi-adapter baselines, offering a simpler, router-free, and more stable solution for multi-task PEFT. The code is available at the anonymous link https://github.com/zhaqian21/CD-LoRA.

Problem

• Multi-Task Learning
• LoRA
• Training-Inference Discrepancy
• Routing Mechanism
• Stability

Innovation

• Consistency-Driven
• Low-Rank Adaptation
• Multi-Task Fine-Tuning
• Training-Inference Discrepancy