Leveraging Audio-LLMs to Filter Speech-to-Speech Training Data

2026-06-11

AI Summary

This work addresses the challenge that large-scale speech-to-speech translation (S2ST) corpora often contain noise, misalignments, and semantic errors, which degrade end-to-end model performance. It proposes a novel unsupervised data filtering approach leveraging audio large language models (Audio-LLMs) through a two-stage Rank-to-Distill strategy. First, an Audio-LLM jointly evaluates acoustic fidelity and cross-lingual semantic consistency to generate pseudo-labels for candidate utterances; these labels then guide the distillation of a high-quality S2ST model. Requiring no human annotations, the method substantially improves translation quality, yielding gains of up to +1.4 ASR-BLEU on the CVSS-C and SpeechMatrix benchmarks.

Abstract

Large-scale mined corpora provide abundant training data for end-to-end speech-to-speech translation (S2ST) but may contain noise, misalignment, and semantic errors. Filtering noisy data is crucial to maintain robust speech translation performance. We study how to train an audio-language model to make keep/drop decisions on paired speech directly from audio. To obtain reliable supervision without manual labels, we adopt a scalable two-stage Rank-to-Distill strategy. A lightweight ranker generates keep/drop pseudo-labels from noisy speech pairs, then trains an audio large language model to predict keep/drop directly from raw paired speech. The resulting model jointly captures acoustic fidelity and cross-lingual semantic consistency for the selection of speech-conditioned data. Experiments on CVSS-C and SpeechMatrix show consistent improvements over unfiltered training, yielding up to +1.4 ASR-BLEU for end-to-end S2ST.

Problem

• speech-to-speech translation
• noisy data filtering
• data quality
• misalignment
• semantic errors

Innovation

• Audio-LLM
• speech-to-speech translation
• data filtering
• Rank-to-Distill
• cross-lingual semantic consistency