Geometric Second-Order Feature Correlation Learning for Self-Supervised Speech Emotion Recognition

2026-06-04

AI Summary

This work addresses a key limitation in existing self-supervised speech emotion recognition methods, which predominantly rely on first-order feature aggregation and thereby neglect higher-order feature correlations and the intrinsic Riemannian geometry of the data, leading to loss of discriminative information. To overcome this, the paper proposes a Second-Order Correlation (SOC) layer that, for the first time, employs covariance matrices as descriptors to model co-occurrence patterns among features. By leveraging Log-Euclidean mapping, the method projects these Riemannian manifold-based descriptors onto a Euclidean tangent space, enabling geometry-preserving linear discriminant learning. This approach relaxes the independence assumption inherent in first-order aggregation and demonstrates significant performance gains over state-of-the-art methods on the ESD and RAVDESS datasets, effectively capturing and exploiting higher-order emotional discriminative cues embedded in self-supervised features.

Abstract

Self-supervised learning (SSL) yields powerful, context-rich representations for speech emotion recognition (SER), yet aggregating these representations into holistic descriptors remains a bottleneck. Conventional first-order aggregation implicitly assumes feature independence, which overlooks the latent Riemannian geometry and discards higher-order relationships essential to the representational power of the backbone. To address this problem, this paper proposes a novel Second-Order Correlation (SOC) layer. Instead of treating features in isolation, SOC models feature correlations as covariance descriptors to capture synergistic co-occurrence patterns, which serve as discriminative signatures for robust emotion recognition. By mapping these descriptors from the Riemannian manifold to a Euclidean tangent space through Log-Euclidean mapping (LEM), the proposed method preserves geometric integrity while enabling direct linear discriminative learning. Extensive experiments on the ESD and RAVDESS datasets demonstrate that SOC recovers discriminative information lost in first-order pooling and effectively aggregates high-dimensional SSL features.

Problem

• self-supervised learning
• speech emotion recognition
• feature aggregation
• Riemannian geometry
• second-order correlation

Innovation

• Second-Order Correlation
• Riemannian Geometry
• Log-Euclidean Mapping
• Self-Supervised Learning
• Speech Emotion Recognition