VoxelFormer: Parameter-Efficient Multi-Subject Visual Decoding from fMRI

2025-09-10

AI Summary

Existing fMRI-based visual decoding methods rely on subject-specific training, suffering from poor generalizability and scalability. To address this, we propose VoxelFormer—a lightweight cross-subject decoding framework. First, it compresses high-dimensional voxel sequences via Token Merging (ToMer); then, a query-driven Q-Former generates fixed-dimensional neural representations explicitly aligned to the CLIP image embedding space, enabling efficient and semantically consistent mapping from fMRI signals to visual reconstructions. With significantly fewer parameters than state-of-the-art (SOTA) methods, VoxelFormer achieves competitive image retrieval performance on the 7T Natural Scenes Dataset. Crucially, it is the first method to demonstrate effective and scalable multi-subject visual reconstruction without incurring substantial parameter overhead, thereby validating both the feasibility and practicality of generalizable fMRI-to-image decoding.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Recent advances in fMRI-based visual decoding have enabled compelling reconstructions of perceived images. However, most approaches rely on subject-specific training, limiting scalability and practical deployment. We introduce extbf{VoxelFormer}, a lightweight transformer architecture that enables multi-subject training for visual decoding from fMRI. VoxelFormer integrates a Token Merging Transformer (ToMer) for efficient voxel compression and a query-driven Q-Former that produces fixed-size neural representations aligned with the CLIP image embedding space. Evaluated on the 7T Natural Scenes Dataset, VoxelFormer achieves competitive retrieval performance on subjects included during training with significantly fewer parameters than existing methods. These results highlight token merging and query-based transformers as promising strategies for parameter-efficient neural decoding.

Problem

• Multi-subject fMRI visual decoding without subject-specific training
• Parameter-efficient neural decoding using lightweight transformer architecture
• Aligning fMRI representations with CLIP image embedding space

Innovation

• Lightweight transformer for multi-subject training
• Token Merging Transformer for voxel compression
• Query-driven Q-Former aligning CLIP embeddings