Study of Lightweight Transformer Architectures for Single-Channel Speech Enhancement

2025-05-27

AI Summary

To address computational constraints and performance bottlenecks in single-channel speech enhancement on edge devices, this paper proposes a lightweight, causal, and real-time model. Methodologically, it introduces: (1) a Frequency-Time-Frequency (FTF) stacking architecture that captures global time-frequency dependencies with minimal parameters; (2) a lightweight Transformer augmented with causal attention to ensure strict causality and ultra-low latency; and (3) a discriminator-assisted adversarial training paradigm to enhance perceptual quality. The model uses only 6% of the parameters of DeepFilterNet2 and reduces MACs by 10% compared to CCFNet+(Lite), with zero additional inference overhead. Evaluated on standard benchmarks, it consistently outperforms existing lightweight models and surpasses several conventional baselines in objective and perceptual metrics.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Natural Language Processing: Speech
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Security and Privacy: Privacy-enhancing technologies
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

In speech enhancement, achieving state-of-the-art (SotA) performance while adhering to the computational constraints on edge devices remains a formidable challenge. Networks integrating stacked temporal and spectral modelling effectively leverage improved architectures such as transformers; however, they inevitably incur substantial computational complexity and model expansion. Through systematic ablation analysis on transformer-based temporal and spectral modelling, we demonstrate that the architecture employing streamlined Frequency-Time-Frequency (FTF) stacked transformers efficiently learns global dependencies within causal context, while avoiding considerable computational demands. Utilising discriminators in training further improves learning efficacy and enhancement without introducing additional complexity during inference. The proposed lightweight, causal, transformer-based architecture with adversarial training (LCT-GAN) yields SoTA performance on instrumental metrics among contemporary lightweight models, but with far less overhead. Compared to DeepFilterNet2, the LCT-GAN only requires 6% of the parameters, at similar complexity and performance. Against CCFNet+(Lite), LCT-GAN saves 9% in parameters and 10% in multiply-accumulate operations yet yielding improved performance. Further, the LCT-GAN even outperforms more complex, common baseline models on widely used test datasets.

Problem

• Achieving SotA speech enhancement with edge device constraints
• Reducing computational complexity in transformer-based spectral modeling
• Lightweight causal transformer architecture for efficient global dependency learning

Innovation

• Streamlined FTF stacked transformers for global dependencies
• Adversarial training enhances learning without inference complexity
• Lightweight causal transformer architecture with minimal overhead