🤖 AI Summary
This study addresses the limited generalization of existing objective speech quality assessment models on unseen data, which hinders their reliable deployment. To this end, the authors propose a Dataset-Split Concealment (DSC) evaluation paradigm, wherein a subset of datasets is withheld during training to simulate real-world unseen conditions and systematically quantify model generalization. Additionally, they introduce a lightweight Aligner module—adding only 1,000 parameters—to effectively mitigate corpus-specific biases. Evaluated across nine training sets and nine unseen test sets, the DSC framework demonstrates consistent improvements when integrated with established models such as MOSNet, NISQA, and Wav2Vec2.0. Notably, the Aligner significantly enhances the accuracy of speech quality estimation on unseen data, particularly for large-scale models like Wav2Vec2.0 (94 million parameters).
📝 Abstract
We introduce Dataset Concealment (DSC), a rigorous new procedure for evaluating and interpreting objective speech quality estimation models. DSC quantifies and decomposes the performance gap between research results and real-world application requirements, while offering context and additional insights into model behavior and dataset characteristics. We also show the benefits of addressing the corpus effect by using the dataset Aligner from AlignNet when training models with multiple datasets. We demonstrate DSC and the improvements from the Aligner using nine training datasets and nine unseen datasets with three well-studied models: MOSNet, NISQA, and a Wav2Vec2.0-based model. DSC provides interpretable views of the generalization capabilities and limitations of models, while allowing all available data to be used at training. An additional result is that adding the 1000 parameter dataset Aligner to the 94 million parameter Wav2Vec model during training does significantly improve the resulting model's ability to estimate speech quality for unseen data.