MedMimic: Physician-Inspired Multimodal Fusion for Early Diagnosis of Fever of Unknown Origin

2025-02-07

AI Summary

Early diagnosis of fever of unknown origin (FUO) remains clinically challenging due to its heterogeneous etiology and nonspecific presentations. Method: We propose a clinician-inspired multimodal fusion framework that jointly models ¹⁸F-FDG PET/CT imaging and structured clinical data—first of its kind for FUO. Our approach employs a learnable self-attention–based Multimodal Fusion Convolutional Network (MFCN), integrating hierarchical features from DINOv2, ViT, and ResNet-18 to overcome limitations of unimodal and shallow fusion methods. Rigorous evaluation is conducted via five-fold cross-validation and systematic ablation studies. Contribution/Results: On a real-world cohort of 416 FUO cases, our model achieves macro-AUROC of 0.8654–0.9291 across seven disease classes—significantly outperforming state-of-the-art machine learning and unimodal deep learning baselines. The framework delivers an interpretable, deployable AI-assisted diagnostic paradigm for precise FUO subtyping, bridging clinical reasoning with multimodal deep learning.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

Fever of unknown origin FUO remains a diagnostic challenge. MedMimic is introduced as a multimodal framework inspired by real-world diagnostic processes. It uses pretrained models such as DINOv2, Vision Transformer, and ResNet-18 to convert high-dimensional 18F-FDG PET/CT imaging into low-dimensional, semantically meaningful features. A learnable self-attention-based fusion network then integrates these imaging features with clinical data for classification. Using 416 FUO patient cases from Sichuan University West China Hospital from 2017 to 2023, the multimodal fusion classification network MFCN achieved macro-AUROC scores ranging from 0.8654 to 0.9291 across seven tasks, outperforming conventional machine learning and single-modality deep learning methods. Ablation studies and five-fold cross-validation further validated its effectiveness. By combining the strengths of pretrained large models and deep learning, MedMimic offers a promising solution for disease classification.

Problem

• Early diagnosis of Fever of Unknown Origin
• Multimodal fusion for medical data
• Improving classification with deep learning

Innovation

• Multimodal fusion for diagnosis
• Self-attention-based feature integration
• Pretrained models for imaging analysis