MOST: MR reconstruction Optimization for multiple downStream Tasks via continual learning

2024-09-16

ISMRM Annual Meeting

AI Summary

Existing MR reconstruction methods prioritize image fidelity while neglecting their impact on downstream tasks (e.g., segmentation, classification), leading to cascaded performance degradation due to error propagation and domain shift. To address this, we propose a continual learning–based reconstruction optimization framework tailored for sequential multi-task deployment. For the first time, we introduce replay-based continual learning into MR reconstruction fine-tuning: a replay buffer jointly optimizes k-space domain reconstruction and downstream task gradients via a multi-task weighted loss, effectively mitigating catastrophic forgetting. Our approach employs a single reconstruction network that concurrently adapts to multiple downstream tasks—preserving high performance across all tasks while eliminating cascade mismatch. Extensive experiments demonstrate that our method significantly outperforms the conventional dual-network paradigm relying on independent optimization.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Search and Optimization: Learning to Search
• Computer Vision: Learning & Optimization for CV

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Motivation: This research aims to address the problem of performance degradation when a reconstruction network and a downstream network are cascaded. The proposed solution, MOST, optimizes a MR reconstruction network for multiple downstream tasks. Goal(s): Our objective is to sequentially finetune a reconstruction network using losses from multiple downstream tasks while preventing catastrophic forgetting such that the same reconstruction network can be used for the multiple tasks. Approach: We introduce replay-based continual learning into finetuning for multiple downstream tasks. Results: Our method successfully circumvents catastrophic forgetting, exhibiting stable performance across all downstream tasks, enabling a single reconstruction network to be used for multiple tasks. Impact: When k-space reconstruction and downstream tasks are performed using two separate networks (individually optimized), the cascade may introduce suboptimal results. Here, we propose a solution when multiple downsteam tasks exist, addressing challenges in realistic user environment.

Problem

• Optimizes MR reconstruction for multiple downstream tasks
• Addresses performance degradation from separate network training
• Uses continual learning to prevent catastrophic forgetting

Innovation

• Continual learning for multiple downstream tasks
• Replay-based and image-guided loss integration
• Optimized MR reconstruction via task-oriented finetuning