Amortizing Federated Adaptation: Hypernetwork Driven LoRA for Personalized Foundation Models

2026-06-04

AI Summary

Existing federated LoRA methods suffer from slow convergence and limited personalization due to structural aggregation bias and delayed client initialization. This work proposes HyperLoRA, the first framework to integrate amortized inference into federated LoRA. It employs a hypernetwork to generate client-specific LoRA initializations and replaces heuristic averaging with learnable aggregation in the low-rank product space. Additionally, a lightweight residual correction mechanism is introduced to enhance training stability. Evaluated under heterogeneous non-IID data settings, HyperLoRA enables unbiased, efficient, and rapidly convergent personalized fine-tuning. It significantly outperforms current approaches across federated vision and vision-language tasks, achieving substantial improvements in convergence speed, robustness, and personalization performance.

Abstract

Federated fine-tuning of foundation models using Low-Rank Adaptation (LoRA) offers a communication efficient solution for distributed learning. However, existing federated LoRA methods suffer from two fundamental limitations: (1) structural aggregation bias, where independently averaging low rank factors fails to approximate the true combined update, and (2) client side initialization lag, as clients repeatedly reinitialize LoRA parameters across communication rounds, slowing convergence. We propose HyperLoRA, a unified framework that addresses both issues through amortized federated adaptation through hypernetwork-driven LoRA generation and product space aggregation. Instead of iterative per-client optimization, HyperLoRA employs a learned generator that maps client distribution signatures to LoRA initializations, effectively amortizing per client adaptation. On the server side, we introduce a learned aggregation module that directly synthesizes updates in the low-rank product space, eliminating the inconsistencies of factor-wise averaging. A lightweight residual correction module further improves stability under heterogenous (non-IID) client distributions.By replacing iterative optimization and heuristic averaging with learned operators, HyperLoRA jointly enables efficient personalization, unbiased aggregation, and faster convergence. Experiments on federated vision and vision-language benchmarks show that HyperLoRA achieves improved convergence speed, greater robustness to distribution shift, and stronger personalization performance compared to prior federated LoRA methods.

Problem

• Federated Learning
• Low-Rank Adaptation
• Structural Aggregation Bias
• Initialization Lag
• Personalized Foundation Models

Innovation

• Hypernetwork
• LoRA
• Federated Learning
• Amortized Adaptation
• Product Space Aggregation