On the Effectiveness of Adaptation Strategies for VLM-Based Federated Learning in Remote Sensing

2026-08-05

AI Summary

This study addresses the challenges of model convergence and degraded generalization in federated learning for remote sensing imagery caused by non-independent and identically distributed (non-IID) client data. It presents the first systematic evaluation of parameter-efficient adaptation strategies for vision-language models (VLMs), including full fine-tuning, encoder-only fine-tuning, prompt learning, and low-rank adaptation (LoRA). Through experiments on BigEarthNet-S2, EuroSAT, RESISC45, and ImageNet, the work analyzes the trade-offs among task-specific performance, cross-domain generalization, communication overhead, and local computational complexity. The findings yield practical guidelines for selecting adaptation strategies under real-world deployment constraints, offering empirical evidence and actionable insights for the efficient application of VLMs in federated remote sensing scenarios.

Abstract

Federated learning (FL) enables collaborative training of deep learning models across decentralized image archives without requiring data centralization. This paradigm is particularly relevant in remote sensing (RS), where legal regulations, privacy concerns, and bandwidth constraints restrict data sharing. However, the presence of training data heterogeneity across clients (known as non-IID data) can impede convergence and limit the generalization capability of the aggregated global model. To mitigate the adverse effects of training data heterogeneity, vision-language models (VLMs) can be leveraged in FL due to their transferable representations, which have demonstrated robustness under distribution shifts. However, their large parameter size may substantially increase communication overhead and local computational complexity in federated settings. Therefore, it is crucial to select an appropriate VLM adaptation strategy that balances the generalization ability with the communication and computational constraints. To address this issue, in this paper, we present the first comparative study of VLM adaptation strategies for FL in the context of RS image classification. We investigate full fine-tuning, encoder-specific fine-tuning, prompt learning, and low-rank adaptation (LoRA) tuning, and analyze them with respect to three criteria: 1) generalization capability under non-IID data, 2) communication overhead, and 3) local computational complexity. Experiments on BigEarthNet-S2, EuroSAT, RESISC45, and ImageNet reveal distinct trade-offs between task specialization, cross-domain generalization, and efficiency. Based on our findings, we derive a guideline for the selection of an appropriate VLM adaptation strategy in FL for RS image classification under different operational constraints. The code of this work is publicly available at https://git.tu-berlin.de/rsim/FL-RS-VLM.

Problem

• federated learning
• vision-language models
• remote sensing
• non-IID data
• adaptation strategies

Innovation

• Vision-Language Models
• Federated Learning
• Parameter-Efficient Fine-Tuning
• Remote Sensing
• Non-IID Data