S-CEReBrO: Breaking the Memory Barrier in Continuous EEG Monitoring

2026-07-30

AI Summary

This work addresses the challenge of applying Transformer models to continuous electroencephalography (EEG) monitoring, where global attention mechanisms incur memory costs that grow linearly with signal duration, hindering scalability to long recordings. To overcome this limitation, the authors propose S-CEReBrO, a novel architecture featuring the first windowed alternating attention mechanism that confines attention computation to fixed spatiotemporal windows, thereby achieving constant key-value (KV) cache memory usage. This approach enables efficient processing of ultra-long EEG sequences, attaining state-of-the-art performance on 7 out of 11 downstream tasks while reducing model parameters by up to 60%. Compared to low-rank linear attention variants, S-CEReBrO cuts memory consumption to 55% and boosts inference throughput by 2.1×, marking the first scalable foundation-model-based framework for continuous EEG monitoring.

Abstract

Foundation models offer a promising paradigm for Electroencephalography (EEG) analysis, leveraging generalizable representations from vast unlabeled datasets. Yet, Transformer-based architectures face a critical bottleneck: global attention mechanisms couple the attention memory state to the signal duration, causing memory overflow during continuous monitoring. To address this, we introduce S-CEReBrO (Streaming CEReBrO), an evolution of the CEReBrO architecture designed for continuous monitoring. Our novel Windowed Alternating Attention mechanism factorizes attention computation into fixed-size spatiotemporal windows, guaranteeing constant KV cache memory as only the active window requires resident attention maps. Empirical scaling analysis confirms that windowed alternating attention can process signals 100X longer than full self-attention and 3X longer than low-rank linear attention. Compared to low-rank linear attention on long contexts, windowed alternating attention requires 55% of the memory while increasing inference throughput by 2.1X. Pre-trained on >25,000 hours of recordings from >12,000 subjects, S-CEReBrO achieves state-of-the-art performance on 7 of 11 downstream tasks, with up to 60% fewer parameters. This work represents a significant step toward the realization of efficient, generalizable, and continuous EEG monitoring. An accompanying code repository is available.

Problem

• continuous EEG monitoring
• memory bottleneck
• Transformer architecture
• attention mechanism
• memory overflow

Innovation

• Windowed Alternating Attention
• Continuous EEG Monitoring
• Memory-Efficient Transformers
• Foundation Models
• KV Cache Optimization