OmniFM: Toward Modality-Robust and Task-Agnostic Federated Learning for Heterogeneous Medical Imaging

2026-03-23

AI Summary

This work addresses the challenges of substantial inter-institutional modality distribution shifts and diverse downstream tasks in medical image analysis under federated learning settings. To this end, we propose OmniFM, the first modality-robust and task-agnostic federated learning framework. OmniFM leverages low-frequency components in the frequency domain to model cross-modal consistency and integrates global spectral knowledge retrieval, embedding-level cross-attention fusion, and a pre-/post-spectral prompt mechanism to uniformly support multiple tasks—including classification, segmentation, super-resolution, and visual question answering—without modifying the training pipeline. The framework jointly models spectral priors and personalized prompts, further enhanced by spectral proximal alignment regularization. Extensive experiments on real-world medical datasets demonstrate that OmniFM consistently outperforms existing methods in both intra- and cross-modality heterogeneous scenarios, under both from-scratch training and fine-tuning setups.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing

Abstract

Federated learning (FL) has become a promising paradigm for collaborative medical image analysis, yet existing frameworks remain tightly coupled to task-specific backbones and are fragile under heterogeneous imaging modalities. Such constraints hinder real-world deployment, where institutions vary widely in modality distributions and must support diverse downstream tasks. To address this limitation, we propose OmniFM, a modality- and task-agnostic FL framework that unifies training across classification, segmentation, super-resolution, visual question answering, and multimodal fusion without re-engineering the optimization pipeline. OmniFM builds on a key frequency-domain insight: low-frequency spectral components exhibit strong cross-modality consistency and encode modality-invariant anatomical structures. Accordingly, OmniFM integrates (i) Global Spectral Knowledge Retrieval to inject global frequency priors, (ii) Embedding-wise Cross-Attention Fusion to align representations, and (iii) Prefix-Suffix Spectral Prompting to jointly condition global and personalized cues, together regularized by a Spectral-Proximal Alignment objective that stabilizes aggregation. Experiments on real-world datasets show that OmniFM consistently surpasses state-of-the-art FL baselines across intra- and cross-modality heterogeneity, achieving superior results under both fine-tuning and training-from-scratch setups.

Problem

• federated learning
• heterogeneous medical imaging
• modality robustness
• task-agnostic
• cross-modality

Innovation

• Federated Learning
• Modality-Robust
• Task-Agnostic
• Spectral Domain
• Medical Imaging