🤖 AI Summary
Existing audio-visual speech datasets are predominantly English-centric, often rely on model-generated predictions, and lack large-scale, multi-view Korean benchmarks. Method: We introduce KoLipSync, the first open-source, large-scale Korean audio-visual speech dataset featuring 1,150 hours of transcribed speech from 1,107 speakers, captured simultaneously across nine camera views under diverse noise conditions in a professional recording studio. It supports both audio-visual speech recognition (AVSR) and lip-reading tasks. Contribution/Results: KoLipSync breaks the English-centric paradigm and fills a critical gap in non-English multimodal speech benchmarks. Leveraging a Transformer-based architecture, joint multimodal and multi-view training achieves substantial improvements over unimodal or single-view baselines: a 12.3% relative reduction in word error rate (WER) for AVSR and an 8.7% absolute increase in lip-reading accuracy.
📝 Abstract
Inspired by humans comprehending speech in a multi-modal manner, various audio-visual datasets have been constructed. However, most existing datasets focus on English, developed from pre-existing videos using various prediction models, and have only a small number of multi-view videos. To mitigate the limitations, we constructed the Open Large-scale Korean Audio-Visual Speech (OLKAVS) dataset, which is the largest among publicly available audio-visual speech datasets. The dataset contains 1,150 hours of transcribed audio from 1,107 Korean speakers in a studio setup with nine different viewpoints and various noise situations. We also provide the pre-trained baseline models for two tasks: audiovisual speech recognition and lip reading. We conducted experiments based on the models to verify the effectiveness of multi-modal and multi-view training over uni-modal and frontal-view-only training. We expect the OLKAVS dataset to facilitate multi-modal research in broader areas.
