Brain-Aligned Multi-Stream Video Transformers with Sparse Self-Selection

2026-07-20

AI Summary

This work addresses the limited neuroscientific grounding and poor biological interpretability of existing video Transformers, which struggle to align with neural activity. Inspired by the primate visual system, the authors propose a dual-stream video Transformer that integrates a high-resolution “what” pathway for object identity with a high-frame-rate “where” pathway for motion and spatial context. The model replaces dense self-attention with a sparse winner-take-all mechanism, incorporating pathway specialization and sparse competition as inductive biases. It achieves Pareto-optimal trade-offs between accuracy and inference efficiency on Kinetics-400 and Something-Something V2, demonstrates enhanced robustness to spatial perturbations, and exhibits a correlation of 0.18 with human EEG neural responses—reaching 78% of the empirical noise ceiling and significantly outperforming mainstream baselines.

Abstract

Modern video transformers typically ignore principles from primate vision and are rarely evaluated against neural data, limiting their biological interpretability. We introduce a sparse winner-takes-all token selection module that replaces dense self-attention to improve efficiency and approximate competitive routing observed in biological visual circuits. We further propose a neuro-inspired split-and-fuse video transformer which uses two complementary pathways: a high-resolution, low-frame-rate "what" stream and a low-resolution, high-frame-rate "where" stream, fused before classification. On Kinetics-400 and Something-Something V2, our best variant operates on the Pareto frontier of accuracy versus inference time among models of comparable scale and pretraining, and showing improved robustness to spatial perturbations. Using representational similarity analysis between model embeddings and time-resolved EEG recordings for the same video stimuli, our model attains a peak brain-model correlation of 0.18 (about 78% of the noise ceiling) and consistently outperforms strong video transformer baselines, suggesting that pathway specialization and sparse competition are useful inductive biases for efficient, brain-aligned video understanding.

Problem

• video transformers
• biological interpretability
• neural data
• primate vision
• brain-aligned

Innovation

• sparse self-selection
• multi-stream video transformer
• neuro-inspired architecture
• representational similarity analysis
• competitive routing