OpenSTBench: Beyond Semantic Evaluation for Speech Translation

2026-05-28

AI Summary

Existing speech translation systems lack a unified multidimensional evaluation protocol, hindering comprehensive comparisons under heterogeneous settings. This work proposes OpenSTBench, the first unified benchmark for both offline and streaming speech-to-text (S2TT) and speech-to-speech (S2ST) translation systems, which standardizes heterogeneous outputs into a common format to enable joint assessment across multiple dimensions—including translation quality, speech fidelity, speaker and emotion preservation, temporal consistency, and latency. Integrating both automatic and human evaluation metrics, the framework spans semantic, acoustic, and temporal aspects. Experimental results reveal that even systems with comparable translation accuracy exhibit significant differences in speech quality and temporal behavior. The project provides reproducible evaluation protocols and open-source tools to support application-oriented, cross-modal system comparisons.

Abstract

Speech translation systems increasingly span speech-to-text translation (S2TT), speech-to-speech translation (S2ST), offline translation, and streaming generation, producing outputs that differ in modality, speech realization, and timing behavior. Existing evaluation practices assess important aspects such as translation quality, speech quality, and temporal quality, but these aspects are often evaluated under separate protocols, making it difficult to compare heterogeneous systems comprehensively. To address this gap, we present OpenSTBench, a unified multidimensional evaluation framework that organizes heterogeneous speech translation outputs into a shared evaluation format. OpenSTBench supports both S2TT and S2ST systems in offline and streaming settings, and jointly evaluates translation quality, speech quality, speaker preservation, emotion and paralinguistic fidelity, temporal consistency, and latency. Through experiments on representative speech translation systems, we show that systems with strong translation quality can still differ substantially in speech quality, as well as in temporal quality. OpenSTBench provides a reproducible protocol for analyzing these cross-dimensional differences and supporting application-oriented comparison of speech translation systems. The code and datasets are available at https://github.com/sjtuayj/OpenSTBench.

Problem

• speech translation
• evaluation framework
• heterogeneous systems
• multidimensional evaluation
• comparative assessment

Innovation

• speech translation evaluation
• multidimensional benchmarking
• S2ST
• temporal consistency
• paralinguistic fidelity