FusionFM: All-in-One Multi-Modal Image Fusion with Flow Matching

2025-11-16

AI Summary

Existing multimodal image fusion methods face three key challenges: poor generalization of task-specific models, slow inference in generative approaches, and scarcity of high-quality ground-truth supervision. To address these, we propose the first flow-matching-based generative framework tailored for multimodal fusion, which directly models a deterministic distributional mapping from source modalities to the fused image—thereby significantly improving sampling efficiency and structural fidelity. Our approach introduces a task-aware pseudo-label filtering mechanism and a Fusion Refiner module, integrated with elastic weight consolidation and experience replay to enable efficient pseudo-supervised learning and continual multi-task adaptation. The resulting lightweight framework achieves state-of-the-art performance on infrared–visible and medical image fusion tasks, with 3.2× faster inference speed, 47% fewer parameters, and markedly enhanced cross-task stability.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

Current multi-modal image fusion methods typically rely on task-specific models, leading to high training costs and limited scalability. While generative methods provide a unified modeling perspective, they often suffer from slow inference due to the complex sampling trajectories from noise to image. To address this, we formulate image fusion as a direct probabilistic transport from source modalities to the fused image distribution, leveraging the flow matching paradigm to improve sampling efficiency and structural consistency. To mitigate the lack of high-quality fused images for supervision, we collect fusion results from multiple state-of-the-art models as priors, and employ a task-aware selection function to select the most reliable pseudo-labels for each task. We further introduce a Fusion Refiner module that employs a divide-and-conquer strategy to systematically identify, decompose, and enhance degraded components in selected pseudo-labels. For multi-task scenarios, we integrate elastic weight consolidation and experience replay mechanisms to preserve cross-task performance and enhance continual learning ability from both parameter stability and memory retention perspectives. Our approach achieves competitive performance across diverse fusion tasks, while significantly improving sampling efficiency and maintaining a lightweight model design. The code will be available at: https://github.com/Ist-Zhy/FusionFM.

Problem

• Unifying multi-modal image fusion tasks with a single model
• Improving sampling efficiency in generative fusion methods
• Addressing lack of supervised data through pseudo-label selection

Innovation

• Leveraging flow matching for efficient image fusion
• Using pseudo-labels from multiple models as supervision
• Employing divide-and-conquer strategy to refine fusion quality