FEDKIM: Adaptive Federated Knowledge Injection into Medical Foundation Models

2024-08-17

Conference on Empirical Methods in Natural Language Processing

AI Summary

Medical foundation models suffer from poor generalization due to multimodal data silos and stringent privacy regulations. To address this, we propose FedKIM, a Federated Knowledge Injection framework that transcends conventional federated learning—where only gradients or model parameters are shared—by enabling privacy-preserving knowledge transfer without raw data transmission. Each participant deploys a lightweight local model to extract private medical knowledge, which is then losslessly injected into a central model via our novel Adaptive Multi-Task Multi-Modal Mixture-of-Experts (M³OE) module. Our key contribution is the first federated knowledge injection paradigm, enabling cross-institutional, cross-modal, and cross-task knowledge fusion. Evaluated on seven modalities and twelve clinical tasks, FedKIM significantly outperforms state-of-the-art baselines, demonstrating concurrent improvements in privacy-compliant generalization and multimodal understanding.

Technology Category

• Machine Learning: Multimodal Learning
• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data
• Computer Vision: Multi-modal Vision

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Foundation models have demonstrated remarkable capabilities in handling diverse modalities and tasks, outperforming conventional artificial intelligence (AI) approaches that are highly task-specific and modality-reliant. In the medical domain, however, the development of comprehensive foundation models is constrained by limited access to diverse modalities and stringent privacy regulations. To address these constraints, this study introduces a novel knowledge injection approach, FedKIM, designed to scale the medical foundation model within a federated learning framework. FedKIM leverages lightweight local models to extract healthcare knowledge from private data and integrates this knowledge into a centralized foundation model using a designed adaptive Multitask Multimodal Mixture Of Experts (M^3OE) module. This method not only preserves privacy but also enhances the model’s ability to handle complex medical tasks involving multiple modalities. Our extensive experiments across twelve tasks in seven modalities demonstrate the effectiveness of FedKIM in various settings, highlighting its potential to scale medical foundation models without direct access to sensitive data. Source codes are available at https://github.com/XiaochenWang-PSU/FedKIM.

Problem

• Adapt federated learning for medical foundation models
• Inject knowledge without violating privacy regulations
• Handle diverse medical modalities and tasks effectively

Innovation

• Federated learning for medical knowledge injection
• Adaptive M3OE module for multimodal integration
• Lightweight local models to preserve privacy