A Unified and Reproducible Experimentation Framework for Speech Understanding

2026-05-29

AI Summary

This work addresses the challenge of incomparable evaluations and irreproducible results in speech understanding models, which often arise from discrepancies in post-processing, data handling, and pipeline design during deployment-oriented model selection. To this end, the authors propose SURE, a unified experimental framework that enables fair evaluation across diverse paradigms—from conventional pipelines to speech large language models—under realistic acoustic and linguistic stressors. SURE achieves this through standardized prediction formats, consistent normalization strategies, and a unified scoring mechanism. Furthermore, it introduces an agent-assisted training conversion pipeline that automatically maps published code into versioned, executable training workflows. This study presents the first unified and reproducible approach for both evaluating and training speech understanding systems across modeling paradigms, substantially enhancing comparability and reproducibility in real-world deployment scenarios.

Abstract

Speech foundation models and Speech LLMs have advanced speech understanding, yet deployment-oriented model selection is hindered by non-comparable evaluations caused by mismatched post-processing, and by training results that are hard to reproduce across data scales and pipelines. We present SURE, a unified experimentation framework that standardizes prediction formats, normalization, and scoring. SURE evaluates strong systems across paradigms, from conventional pipelines to Speech LLMs, on representative tasks under realistic acoustic and linguistic stressors. Beyond evaluation, SURE introduces an agent-assisted training conversion flow that maps paper and code into versioned, runnable training pipelines under a unified protocol on matched open-data subsets. Overall, SURE improves comparability and reproducibility for deployment-oriented evaluation.

Problem

• speech understanding
• model evaluation
• reproducibility
• comparability
• deployment-oriented selection

Innovation

• unified evaluation framework
• reproducible training pipeline
• speech foundation models
• agent-assisted conversion
• standardized scoring