Beyond Residual Connections: Manifold-Constrained Hyper-Connections for Robust Speaker Representation Learning

2026-08-05

AI Summary

This work addresses the limitations of conventional residual connections, which employ identity mappings that restrict information flow and hinder speaker representation capacity. To overcome this, the authors propose manifold-constrained hyper-connections (mHC), reformulating residual structures into a multi-stream evolution mechanism. The approach integrates information from multiple pathways via doubly stochastic matrices and enforces manifold constraints through Sinkhorn–Knopp iterations to preserve signal magnitude and feature mean, thereby ensuring energy conservation and gradient stability. The mHC framework seamlessly integrates into mainstream architectures such as ECAPA-TDNN and ResNet, yielding significant performance gains in speaker verification on VoxCeleb1, which demonstrates its generality, robustness, and effectiveness.

Abstract

Residual connections are fundamental to deep speaker recogni- tion models, such as ECAPA-TDNN and ResNet. However, standard identity mapping limits information flow to a sin- gle path, constraining representation capacity. We introduce Manifold-Constrained Hyper-Connections (mHC), reformulat- ing residual paths as a multi-stream evolution where informa- tion is mixed through a doubly stochastic matrix. By employing Sinkhorn-Knopp iterations, mHC ensures energy conservation by preserving signal intensity and feature mean, which stabi- lizes gradients and mitigates signal degradation in complex net- works. We evaluate mHC by replacing standard residual con- nections in backbones including ECAPA-TDNN, ResNet-34, Res2Net, and E-Res2Net. Extensive experiments on VoxCeleb1 demonstrate that mHC connections consistently enhance per- formance across all architectures, highlighting its effectiveness for robust speaker representation learning.

Problem

• residual connections
• speaker representation learning
• information flow
• representation capacity
• signal degradation

Innovation

• Manifold-Constrained Hyper-Connections
• doubly stochastic matrix
• Sinkhorn-Knopp iterations
• residual connections
• speaker representation learning