Multilingual Long-Form Speech Instruction Following: KIT's Submission to IWSLT 2026

2026-06-03

AI Summary

This work addresses key challenges in multilingual long-form spoken language understanding—namely, unknown task specifications, susceptibility to overfitting, and semantic inconsistency—by proposing a unified data augmentation pipeline. The approach constructs over one million multilingual, multitask training samples through concatenation of short speech segments, enhanced by large language model (LLM) prompting and cross-lingual translation, with LLM-assisted annotation to ensure label quality. To overcome the limitations of conventional likelihood-based rescoring in semantic tasks, the method further integrates minimum Bayes risk decoding, substantially improving holistic semantic comprehension of long utterances. Evaluated on the IWSLT 2026 unconstrained instruction-following track, the proposed framework significantly outperforms baseline systems while effectively mitigating semantic distortion caused by segment-wise processing.

Abstract

With the advent of Large Language Models, single-task and token-based multi-task models have evolved into instruction-based systems that infer task and target language implicitly from natural language prompts. This trend is reflected in IWSLT's Instruction Following Track, which this year introduced new tasks including an unknown surprise task, posing a genuine challenge against overfitting to known tasks. We present KIT's submission to the Long and Short Instruction Following tracks in the unconstrained setting. Our approach combines a general data augmentation pipeline that converts short-form corpora into long-form training data through segment concatenation, LLM-based label generation, and cross-lingual translation, yielding over 1M instances across six tasks and four languages. We further show that likelihood-based re-ranking, while highly effective for ASR, systematically degrades semantic tasks by spuriously selecting candidates generated from segmented audio processing rather than holistic long-form inference, a failure mode resolved by combining likelihood with Minimum Bayes Risk decoding.

Problem

• instruction following
• multilingual
• long-form speech
• overfitting
• semantic tasks

Innovation

• instruction following
• long-form speech
• data augmentation
• Minimum Bayes Risk decoding
• multilingual