ThanoRA: Task Heterogeneity-Aware Multi-Task Low-Rank Adaptation

2025-05-24

AI Summary

To address task heterogeneity loss, subspace interference and collapse, and inference overhead induced by parameter merging in multi-task LoRA, this paper proposes the first low-rank adaptation framework that jointly preserves task heterogeneity and subspace independence. Our method introduces: (1) a task-aware LoRA subspace initialization mechanism that explicitly models inter-task differences; (2) an orthogonality regularization constraint on LoRA subspaces to suppress cross-task interference; and (3) a joint training paradigm featuring full parameter sharing, routing-free design, and mergeable adapters. Crucially, the approach incurs zero inference overhead and introduces no additional parameters. Extensive experiments demonstrate that it consistently outperforms strong baselines—including MoE-LoRA—across both multimodal and pure-text multi-task benchmarks, achieving significant gains in generalization performance and task synergy efficiency.

Technology Category

Machine Learning: Mixture of Experts (MoE)Computer Vision: Multi-modal VisionNatural Language Processing: Language Grounding & Multi-modal NLP

Application Category

Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingUser Modeling, Personalization and Recommendation: Fairness-aware retrieval and rankingSemantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Low-Rank Adaptation (LoRA) is widely adopted for downstream fine-tuning of foundation models due to its efficiency and zero additional inference cost. Many real-world applications require foundation models to specialize in multiple tasks simultaneously, motivating the need for efficient multi-task adaptation. While recent approaches integrate LoRA with mixture-of-experts (MoE) to address this, the use of routers prevents parameter mergeability, which increases inference overhead and hinders unified multi-task adaptation, thereby limiting deployment practicality. In this work, we propose ThanoRA, a Task Heterogeneity-Aware Multi-Task Low-Rank Adaptation framework that enables multi-task adaptation while preserving the inference efficiency of LoRA. ThanoRA jointly models task heterogeneity and mitigates subspace interference throughout training. Specifically, motivated by inherent differences in complexity and heterogeneity across tasks, ThanoRA constructs task-specific LoRA subspaces at initialization, enabling fine-grained knowledge injection aligned with task heterogeneity. Furthermore, to prevent task interference and subspace collapse during multi-task training, ThanoRA introduces a subspace-preserving regularization that maintains the independence of task-specific representations. With the synergy of both components, ThanoRA enables efficient and unified multi-task adaptation. Extensive experiments across multimodal and text-only benchmarks under varying multi-task mixtures demonstrate that ThanoRA consistently achieves robust and superior performance over strong baselines without introducing additional inference overhead. Our code is publicly available at: https://github.com/LiangJian24/ThanoRA.

Problem

• Enables multi-task adaptation preserving LoRA inference efficiency
• Mitigates subspace interference during multi-task training
• Addresses task heterogeneity with specialized LoRA subspaces

Innovation

• Task-specific LoRA subspaces for heterogeneity
• Subspace-preserving regularization prevents interference
• Unified multi-task adaptation without overhead