A Stage-Wise Learning Strategy with Fixed Anchors for Robust Speaker Verification

2025-10-21

AI Summary

To address the challenge of simultaneously achieving discriminability and noise robustness in speaker representation learning under noisy conditions, this paper proposes a fixed-anchor-based two-stage learning framework. In the first stage, a base model is trained on clean speech data to construct highly discriminative speaker anchor representations. In the second stage, the model is fine-tuned on noisy data, where anchor-distance regularization constrains the feature space to decouple discriminative boundary stabilization from noise-induced variation suppression. The fixed anchors serve as invariant references, effectively preventing feature drift caused by joint optimization. Extensive experiments across diverse noise conditions demonstrate that the proposed method significantly outperforms end-to-end joint-optimization baselines: it preserves strong speaker discriminability while substantially improving robustness. This work establishes a novel paradigm for noise-robust speaker verification.

Technology Category

Machine Learning: Adversarial Learning & RobustnessNatural Language Processing: Safety and RobustnessIntelligent Robots: Learning & Optimization for ROB

Application Category

Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingWeb Mining and Content Analysis: Robustness and generalizability of Web mining methodsSemantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Learning robust speaker representations under noisy conditions presents significant challenges, which requires careful handling of both discriminative and noise-invariant properties. In this work, we proposed an anchor-based stage-wise learning strategy for robust speaker representation learning. Specifically, our approach begins by training a base model to establish discriminative speaker boundaries, and then extract anchor embeddings from this model as stable references. Finally, a copy of the base model is fine-tuned on noisy inputs, regularized by enforcing proximity to their corresponding fixed anchor embeddings to preserve speaker identity under distortion. Experimental results suggest that this strategy offers advantages over conventional joint optimization, particularly in maintaining discrimination while improving noise robustness. The proposed method demonstrates consistent improvements across various noise conditions, potentially due to its ability to handle boundary stabilization and variation suppression separately.

Problem

• Learning robust speaker representations under noisy conditions
• Establishing discriminative speaker boundaries with stable anchor embeddings
• Maintaining speaker identity while improving noise robustness

Innovation

• Stage-wise learning with fixed anchor embeddings
• Base model training for discriminative speaker boundaries
• Fine-tuning with anchor regularization for noise robustness