SoCov: Semi-Orthogonal Parametric Pooling of Covariance Matrix for Speaker Recognition

2025-04-06

IEEE International Conference on Acoustics, Speech, and Signal Processing

AI Summary

Traditional frame-level pooling (e.g., mean + standard deviation) neglects inter-feature covariances, leading to loss of speaker-discriminative information. To address this, we propose SoCov pooling: first, frame features are enhanced via self-attention to construct a robust covariance matrix; second, a semi-orthogonal parametrized vectorization mechanism is introduced to enable stable, differentiable compression of the covariance structure; finally, a weighted standard deviation vector is fused to produce segment-level sc-vector embeddings. This work constitutes the first end-to-end differentiable framework that explicitly models and optimizes the full covariance structure within deep speaker embedding architectures. Evaluated on the SRE21 evaluation set, the sc-vector achieves a 15.5% relative reduction in equal error rate (EER) over x-vectors and a 30.9% relative improvement over conventional pooling methods, demonstrating substantial gains in both speaker verification accuracy and robustness.

Technology Category

• Machine Learning: Semi-Supervised Learning
• Search and Optimization: Non-convex Optimization
• Computer Vision: Learning & Optimization for CV

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

In conventional deep speaker embedding frameworks, the pooling layer aggregates all frame-level features over time and computes their mean and standard deviation statistics as inputs to subsequent segment-level layers. Such statistics pooling strategy produces fixed-length representations from variable-length speech segments. However, this method treats different frame-level features equally and discards covariance information. In this paper, we propose the Semi-orthogonal parameter pooling of Covariance matrix (SoCov) method. The SoCov pooling computes the covariance matrix from the self-attentive frame-level features and compresses it into a vector using the semi-orthogonal parametric vectorization, which is then concatenated with the weighted standard deviation vector to form inputs to the segment-level layers. Deep embedding based on SoCov is called ``sc-vector''. The proposed sc-vector is compared to several different baselines on the SRE21 development and evaluation sets. The sc-vector system significantly outperforms the conventional x-vector system, with a relative reduction in EER of 15.5% on SRE21Eval. When using self-attentive deep feature, SoCov helps to reduce EER on SRE21Eval by about 30.9% relatively to the conventional ``mean + standard deviation'' statistics.

Problem

• Improving speaker recognition by capturing covariance information
• Replacing conventional statistics pooling with semi-orthogonal parametric pooling
• Enhancing deep speaker embedding through self-attentive feature compression

Innovation

• Semi-orthogonal parametric pooling of covariance
• Self-attentive frame-level features compression
• Concatenation with weighted standard deviation vector