CineBrain: A Large-Scale Multi-Modal Brain Dataset During Naturalistic Audiovisual Narrative Processing

2025-03-10

AI Summary

This work addresses the bottleneck in spatiotemporal decoding of brain signals under natural audiovisual stimulation—specifically, how to jointly leverage EEG’s millisecond-scale temporal resolution and fMRI’s whole-brain spatial coverage. To this end, we introduce the first large-scale synchronized EEG-fMRI dataset, comprising 36 hours of recordings from six participants watching *The Big Bang Theory*. We propose CineSync, a novel framework integrating multimodal encoders, a diffusion-based neural latent variable decoder, and an EEG-fMRI joint modeling mechanism. Furthermore, we pioneer a cross-modal alignment strategy tailored for audiovisual reconstruction and establish Cine-Benchmark—a comprehensive, multidimensional evaluation benchmark. Experiments demonstrate state-of-the-art performance on video reconstruction tasks and, for the first time, empirically validate the feasibility and efficacy of synergistic EEG-fMRI decoding for complex natural stimuli.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

In this paper, we introduce CineBrain, the first large-scale dataset featuring simultaneous EEG and fMRI recordings during dynamic audiovisual stimulation. Recognizing the complementary strengths of EEG's high temporal resolution and fMRI's deep-brain spatial coverage, CineBrain provides approximately six hours of narrative-driven content from the popular television series The Big Bang Theory for each of six participants. Building upon this unique dataset, we propose CineSync, an innovative multimodal decoding framework integrates a Multi-Modal Fusion Encoder with a diffusion-based Neural Latent Decoder. Our approach effectively fuses EEG and fMRI signals, significantly improving the reconstruction quality of complex audiovisual stimuli. To facilitate rigorous evaluation, we introduce Cine-Benchmark, a comprehensive evaluation protocol that assesses reconstructions across semantic and perceptual dimensions. Experimental results demonstrate that CineSync achieves state-of-the-art video reconstruction performance and highlight our initial success in combining fMRI and EEG for reconstructing both video and audio stimuli. Project Page: https://jianxgao.github.io/CineBrain.

Problem

• Integrates EEG and fMRI for audiovisual stimulus reconstruction.
• Introduces CineBrain dataset with narrative-driven EEG and fMRI recordings.
• Proposes CineSync framework for multimodal signal fusion and decoding.

Innovation

• Combines EEG and fMRI for brain activity analysis
• Introduces CineSync for multimodal signal fusion
• Develops Cine-Benchmark for evaluation protocol