Time-Frequency-Based Attention Cache Memory Model for Real-Time Speech Separation

2025-05-19

AI Summary

To address the performance degradation of causal speech separation models caused by historical information forgetting, this paper proposes the Time-Frequency Attention Cache Memory (TFACM) architecture—the first to jointly optimize time-frequency modeling and lightweight cache memory. Methodologically, TFACM integrates LSTM-based frequency-domain modeling, causal time-domain local and global representations, and a Causal Attention Refinement (CAR) module, thereby enhancing long-range temporal dependency modeling under strict causality constraints. Experiments on WSJ0-2mix demonstrate a 0.18 PESQ improvement with real-time latency under 30 ms. Compared to the state-of-the-art TF-GridNet-Causal, TFACM reduces computational complexity significantly and decreases trainable parameters by 42%, achieving an effective trade-off among low latency, high fidelity, and efficiency.

Technology Category

• Machine Learning: Causal Learning
• Cognitive Modeling & Cognitive Systems: Affective Computing
• Natural Language Processing: Speech

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Existing causal speech separation models often underperform compared to non-causal models due to difficulties in retaining historical information. To address this, we propose the Time-Frequency Attention Cache Memory (TFACM) model, which effectively captures spatio-temporal relationships through an attention mechanism and cache memory (CM) for historical information storage. In TFACM, an LSTM layer captures frequency-relative positions, while causal modeling is applied to the time dimension using local and global representations. The CM module stores past information, and the causal attention refinement (CAR) module further enhances time-based feature representations for finer granularity. Experimental results showed that TFACM achieveed comparable performance to the SOTA TF-GridNet-Causal model, with significantly lower complexity and fewer trainable parameters. For more details, visit the project page: https://cslikai.cn/TFACM/.

Problem

• Improves real-time speech separation performance
• Retains historical information using cache memory
• Reduces model complexity and parameters

Innovation

• Time-Frequency Attention Cache Memory model
• LSTM layer captures frequency-relative positions
• Causal attention refinement enhances feature representations