Flow of Knowledge: Federated Fine-Tuning of LLMs in Healthcare under Non-IID Conditions

2025-10-01

AI Summary

To address model convergence instability, poor fairness, and privacy compliance challenges arising from medical data silos and cross-institutional non-IID data distributions, this paper proposes a privacy-preserving federated fine-tuning framework. The method integrates Low-Rank Adaptation (LoRA) for efficient, lightweight local updates; enhances knowledge transfer across heterogeneous clients via an improved Federated Averaging algorithm; and incorporates blockchain to enable decentralized identity authentication and auditable model aggregation. Evaluated on highly non-IID real-world medical text data, the framework significantly improves convergence stability, boosts cross-client generalization by 23.6%, and increases average performance of the worst-performing clients by 41.2%. Crucially, it ensures strict data locality (no raw data leaves client premises) and end-to-end traceability of collaborative training. This work establishes a scalable, fair, and regulatory-compliant technical pathway for multi-center collaborative training of large language models in healthcare.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Natural Language Processing: Safety and Robustness
• Humans and AI: Human-in-the-loop Machine Learning

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Data transparency and provenance

Abstract

Large language models (LLMs) show great promise in healthcare, but their applications are hindered by data privacy restrictions and the challenges of cross-institution collaboration. Sensitive medical data cannot be centralized, while non-independent and identically distributed (non-IID) characteristics across institutions further com- plicate convergence and fairness. To address these issues, we present a federated fine-tuning approach based on Low-Rank Adaptation (LoRA), enabling privacy-preserving knowledge flow across institu- tions. The method iteratively combines local LoRA adaptation with global parameter aggregation, allowing efficient knowledge shar- ing without exposing raw data. A blockchain identity scheme is used for identifying individual LLM in such a distributed network. We evaluate this approach on heterogeneous and highly non-IID medical text datasets, where experiments demonstrate that feder- ated LoRA not only enhances cross-client generalization but also improves the performance of the weakest client, achieving stable convergence and fairer outcomes. These findings highlight federated LoRA fine-tuning as a practical and effective paradigm for adapting LLMs in healthcare, offering a new path for multi-center medical AI collaboration.

Problem

• Addressing data privacy and non-IID challenges in healthcare LLM applications
• Enabling privacy-preserving knowledge sharing across institutions without raw data exposure
• Improving cross-client generalization and fairness in federated medical AI systems

Innovation

• Federated fine-tuning using LoRA for privacy
• Local LoRA adaptation with global aggregation
• Blockchain identity for distributed LLM identification