FedWeave: Rethinking the Unit of Specialization in Heterogeneous Federated MoE-LoRA

2026-07-29

AI Summary

This work addresses the challenges of cross-task interference and gradient conflicts arising from client task heterogeneity in federated learning by proposing FedWeave, a novel framework that introduces an asymmetric aggregation mechanism. FedWeave is the first to explicitly enforce “purity” in expert modules and “contrastiveness” in routing components. It achieves prototype-level expert specialization through unsupervised prototype discovery and cross-client alignment, while leveraging MoE-LoRA with sparse activation for efficient multi-task routing. Evaluated on mainstream large language models and heterogeneous multitask benchmarks, FedWeave significantly outperforms existing methods. Theoretical analysis further corroborates the efficacy of the proposed asymmetric aggregation strategy.

Abstract

Federated PEFT enables LLMs to collaboratively adapt to decentralized private data without sharing raw examples. However, task heterogeneity across clients can cause cross-task interference and gradient conflicts during aggregation. Federated MoE-LoRA addresses this challenge through specialized LoRA experts and conditional routing. Yet existing methods typically specialize at client granularity, implicitly assuming task-coherent clients. Our core insight is that experts need purity, namely pattern-coherent updates that preserve specialization, whereas routers need contrast, namely mixed-task observations that support expert comparison. We propose FedWeave, a framework that adopts asymmetric aggregation, separating expert aggregation from router optimization to meet these two requirements. FedWeave uses unsupervised prototype discovery to form local buckets and align them across clients, enabling prototype-level expert aggregation while retaining mixed-task client trajectories for router training. At inference, FedWeave performs sparse inference with one active expert while preserving nearly all soft-routing performance. Our theoretical analysis explains why asymmetric aggregation is advantageous: it controls expert convergence in stationarity through off-pattern contamination, identifies the consensus error induced by fragmented router trajectories, and bounds sparse-inference risk. On a heterogeneous multi-task benchmark with mainstream LLM backbones, FedWeave consistently outperforms strong baselines, while ablations verify the effectiveness of our design.

Problem

• federated learning
• task heterogeneity
• MoE-LoRA
• cross-task interference
• expert specialization

Innovation

• Federated MoE-LoRA
• asymmetric aggregation
• prototype discovery
• task heterogeneity
• sparse inference