Federated Foundation Models Fine-Tuning with Heterogeneous Compressed Clients

2026-07-31

AI Summary

This work addresses the challenge of deploying large foundation models on resource-constrained clients in federated learning by proposing FedSLM, a novel framework that constructs self-contained lightweight client models via SVD-based low-rank decomposition. FedSLM introduces a two-stage aggregation protocol: intra-group synchronization of low-rank adapters followed by inter-group fusion of full-rank representations. The method innovatively incorporates a structure-aligned fusion mechanism operating on nested subspace manifolds, complemented by a confidence-guided auxiliary loss and a weak-to-strong knowledge distillation strategy to enable efficient knowledge transfer. Experimental results demonstrate that FedSLM significantly outperforms existing federated approaches on both natural language and vision-language tasks, maintaining strong representation capabilities under both IID and non-IID data settings while reducing client memory consumption to approximately 50% of that required by the full model.

Abstract

Federated learning of foundation models faces a fundamental resource-asymmetry challenge: the institutions holding the most valuable domain-specific data cannot host billion-parameter models. Existing heterogeneous federated approaches attempt to bridge this gap through parameter-efficient tuning, model pruning, or knowledge distillation, yet each trades away a critical property, whether full-model memory reduction, architectural self-containedness, or representational fidelity, leaving the core tension unresolved. We propose FedSLM, a parameter-centric framework for federated fine-tuning with heterogeneous compressed clients. FedSLM uses SVD-based decomposition to produce self-contained client models, whose low-rank subspaces form nested manifolds that are structurally compatible for aggregation. It then applies a two-stage protocol that synchronizes lightweight adapters within compression groups and fuses full-rank reconstructions across groups via structural alignment. Finally, a weak-to-strong elicitation step with auxiliary confidence loss transfers the aggregated knowledge to the full-scale server, while an explicit bias--variance trade-off mitigates compression artifacts. We provide theoretical guarantees for adapter-level aggregation, subspace-alignment bounds for cross-group fusion, and a characterization of how the confidence loss mitigates weak-supervision noise. Experiments on natural language and vision--language benchmarks show that FedSLM outperforms existing federated baselines under both IID and non-IID partitions, while client models operate at roughly 50% of the GPU memory required by the full model.

Problem

• federated learning
• foundation models
• resource asymmetry
• heterogeneous clients
• model compression

Innovation

• Federated Learning
• Foundation Models
• Heterogeneous Clients
• Low-Rank Decomposition
• Parameter-Efficient Tuning