Shift-Dependent Asymmetry: Orthogonal Inverse Low-Rank Adaptation for Federated Medical Segmentation

2026-06-07

AI Summary

Existing federated LoRA approaches employ a uniform aggregation strategy that fails to account for the heterogeneous nature of medical image segmentation, where encoder updates are dominated by appearance shifts while decoder updates are driven by site-specific supervision discrepancies. This conflation of shared anatomical structures and local biases compromises model generalization. To address this, we propose Inverse Asymmetric Tuning (IAT), which introduces module-specific LoRA adapters in the encoder and decoder to absorb appearance variations and adapt to local supervisory signals, respectively, while preserving a shared pathway for consensus knowledge propagation. Furthermore, we design a subspace orthogonality regularizer that effectively decouples shared and local updates under low-rank parameterization without incurring additional communication overhead. By explicitly modeling encoder–decoder heterogeneity as a module-adaptive problem, IAT achieves state-of-the-art performance across multiple federated medical segmentation benchmarks, significantly enhancing generalization.

Abstract

Low-Rank Adaptation (LoRA) enables efficient federated fine-tuning of segmentation foundation models for medical imaging. However, most federated LoRA methods adopt a uniform aggregation rule, which breaks under the encoder-decoder asymmetry in medical segmentation: the encoder is dominated by appearance shifts, while the decoder is dominated by supervision variations. This mismatch entangles shared anatomy with site-specific biases and harms generalization. To address this, we propose Inverse Asymmetric Tuning (IAT). IAT aligns adaptation with heterogeneity sources by personalizing module-specific components in the encoder to absorb appearance shifts and in the decoder to accommodate site-dependent supervision, while retaining a shared pathway for transferable consensus. However, structural separation alone is insufficient under LoRA's bilinear parameterization, where multiplicative coupling can still cause site-specific updates to leak into the shared direction. We therefore introduce a Subspace Orthogonality Regularizer that penalizes shared-local collinearity in the effective update space, mitigating leakage without extra communication. Experiments show consistent improvements over strong federated LoRA and parameter-efficient FL baselines.

Problem

• Federated Learning
• Medical Segmentation
• Low-Rank Adaptation
• Domain Shift
• Model Asymmetry

Innovation

• Inverse Asymmetric Tuning
• Subspace Orthogonality Regularizer
• Federated Medical Segmentation
• Low-Rank Adaptation
• Encoder-Decoder Asymmetry