Brain3D: EEG-to-3D Decoding of Visual Representations via Multimodal Reasoning

2026-04-09

AI Summary

This work addresses the limitation of existing EEG decoding approaches, which are largely confined to reconstructing two-dimensional images and struggle to recover three-dimensional visual representations, thereby hindering geometric understanding and practical deployment. To overcome this, the authors propose a multi-stage, geometry-aware end-to-end framework that first decodes EEG signals into 2D images, then leverages a multimodal large language model to generate structured 3D descriptions, which subsequently guide a diffusion model to synthesize coherent 3D content and produce consistent 3D meshes. This approach establishes the first complete decoding pipeline from EEG to 3D representations, circumventing the intractability and lack of scalability associated with direct mapping strategies. Experimental results across ten object categories demonstrate a Top-1 accuracy of 85.4% and a CLIPScore of 0.648, confirming the feasibility and efficacy of EEG-driven 3D reconstruction.

Abstract

Decoding visual information from electroencephalography (EEG) has recently achieved promising results, primarily focusing on reconstructing two-dimensional (2D) images from brain activity. However, the reconstruction of three-dimensional (3D) representations remains largely unexplored. This limits the geometric understanding and reduces the applicability of neural decoding in different contexts. To address this gap, we propose Brain3D, a multimodal architecture for EEG-to-3D reconstruction based on EEG-to-image decoding. It progressively transforms neural representations into the 3D domain using geometry-aware generative reasoning. Our pipeline first produces visually grounded images from EEG signals, then employs a multimodal large language model to extract structured 3D-aware descriptions, which guide a diffusion-based generation stage whose outputs are finally converted into coherent 3D meshes via a single-image-to-3D model. By decomposing the problem into structured stages, the proposed approach avoids direct EEG-to-3D mappings and enables scalable brain-driven 3D generation. We conduct a comprehensive evaluation comparing the reconstructed 3D outputs against the original visual stimuli, assessing both semantic alignment and geometric fidelity. Experimental results demonstrate strong performance of the proposed architecture, achieving up to 85.4% 10-way Top-1 EEG decoding accuracy and 0.648 CLIPScore, supporting the feasibility of multimodal EEG-driven 3D reconstruction.

Problem

• EEG-to-3D decoding
• visual representation
• 3D reconstruction
• neural decoding
• multimodal reasoning

Innovation

• EEG-to-3D decoding
• multimodal reasoning
• geometry-aware generation
• diffusion-based 3D reconstruction
• neural decoding