Many-Task Federated Fine-Tuning via Unified Task Vectors

2025-02-10

AI Summary

To address the challenge of strong task heterogeneity among clients in multi-task federated learning (MaT-FL), where existing approaches rely on server-side client grouping or personalized models and struggle to support client-autonomous, diverse downstream tasks, this paper proposes a novel MaT-FL framework. Methodologically, it introduces: (1) a globally shared task vector, enabling cross-client joint learning of task representations via a direction-similarity-driven aggregation mechanism—eliminating the need for server-side clustering or client-specific parameter storage; and (2) a lightweight learnable modulator that facilitates knowledge transfer while preserving task disentanglement. Extensive experiments across 30 benchmark datasets demonstrate that the method achieves performance on par with single-task fine-tuning, significantly outperforms state-of-the-art MaT-FL baselines, and reduces communication overhead substantially.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Multiagent Systems: Multiagent Learning
• Computer Vision: Multi-modal Vision

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Federated Learning (FL) traditionally assumes homogeneous client tasks; however, in real-world scenarios, clients often specialize in diverse tasks, introducing task heterogeneity. To address this challenge, Many-Task FL (MaT-FL) has emerged, enabling clients to collaborate effectively despite task diversity. Existing MaT-FL approaches rely on client grouping or personalized layers, requiring the server to manage individual models and failing to account for clients handling multiple tasks. We propose MaTU, a MaT-FL approach that enables joint learning of task vectors across clients, eliminating the need for clustering or client-specific weight storage at the server. Our method introduces a novel aggregation mechanism that determines task similarity based on the direction of clients task vectors and constructs a unified task vector encapsulating all tasks. To address task-specific requirements, we augment the unified task vector with lightweight modulators that facilitate knowledge transfer among related tasks while disentangling dissimilar ones. Evaluated across 30 datasets, MaTU achieves superior performance over state-of-the-art MaT-FL approaches, with results comparable to per-task fine-tuning, while delivering significant communication savings.

Problem

• Addresses task heterogeneity in Federated Learning
• Proposes unified task vectors for joint learning
• Enhances knowledge transfer with lightweight modulators

Innovation

• Unified Task Vector Learning
• Task Similarity Aggregation
• Lightweight Modulator Augmentation