COBRA: A Continual Learning Approach to Vision-Brain Understanding

2024-11-25

arXiv.org

AI Summary

To address catastrophic forgetting in fMRI-driven vision–brain understanding (VBU) under cross-subject continual learning, this paper proposes a dual-module framework integrating Subject Commonality and Prompt-based Subject Specificity—marking the first approach to decouple universal brain response representation from subject-specific modeling. Built upon MRIFormer, a Transformer architecture specifically designed for fMRI, our method incorporates learnable prompts, cross-subject shared feature extraction, and modular incremental training. Evaluated on both continual learning and visual stimulus reconstruction tasks, it achieves state-of-the-art performance: average forgetting rate decreases by 42%, and peak signal-to-noise ratio (PSNR) in reconstruction improves by 2.1 dB. These results demonstrate superior generalizability and adaptability in neural decoding, effectively balancing knowledge retention across subjects with individualized functional mapping.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Vision-Brain Understanding (VBU) aims to extract visual information perceived by humans from brain activity recorded through functional Magnetic Resonance Imaging (fMRI). Despite notable advancements in recent years, existing studies in VBU continue to face the challenge of catastrophic forgetting, where models lose knowledge from prior subjects as they adapt to new ones. Addressing continual learning in this field is, therefore, essential. This paper introduces a novel framework called Continual Learning for Vision-Brain (COBRA) to address continual learning in VBU. Our approach includes three novel modules: a Subject Commonality (SC) module, a Prompt-based Subject Specific (PSS) module, and a transformer-based module for fMRI, denoted as MRIFormer module. The SC module captures shared vision-brain patterns across subjects, preserving this knowledge as the model encounters new subjects, thereby reducing the impact of catastrophic forgetting. On the other hand, the PSS module learns unique vision-brain patterns specific to each subject. Finally, the MRIFormer module contains a transformer encoder and decoder that learns the fMRI features for VBU from common and specific patterns. In a continual learning setup, COBRA is trained in new PSS and MRIFormer modules for new subjects, leaving the modules of previous subjects unaffected. As a result, COBRA effectively addresses catastrophic forgetting and achieves state-of-the-art performance in both continual learning and vision-brain reconstruction tasks, surpassing previous methods.

Problem

• Addresses catastrophic forgetting in vision-brain understanding
• Introduces COBRA for continual learning in VBU
• Combines shared and unique vision-brain patterns

Innovation

• COBRA framework for continual learning
• Transformer-based fMRI feature extraction
• Modules to mitigate catastrophic forgetting