Scaling and Prompting for Improved End-to-End Spoken Grammatical Error Correction

2025-05-27

AI Summary

End-to-end spoken grammar error correction and feedback generation (SGECF) suffers from performance bottlenecks due to severe scarcity of annotated data. Method: We propose a joint optimization framework integrating pseudo-labeling and instruction-based prompting: (i) Whisper is employed for end-to-end speech-to-text mapping; (ii) pseudo-labeling expands the training set to 2500 hours; (iii) fluent transcriptions serve as contextual prompts to guide feedback generation and fine-tuning. Contributions/Results: This work is the first to synergistically combine pseudo-labeling and instruction prompting for end-to-end SGECF. It empirically uncovers non-monotonic interactions among model scale, data expansion, and prompting strategies. Crucially, it demonstrates that instruction prompting consistently benefits large language models, whereas pseudo-labeling gains diminish with increasing model size. Experiments show substantial improvements in grammatical error correction accuracy and feedback quality, establishing a novel paradigm for low-resource spoken language understanding.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Spoken Grammatical Error Correction (SGEC) and Feedback (SGECF) are crucial for second language learners, teachers and test takers. Traditional SGEC systems rely on a cascaded pipeline consisting of an ASR, a module for disfluency detection (DD) and removal and one for GEC. With the rise of end-to-end (E2E) speech foundation models, we investigate their effectiveness in SGEC and feedback generation. This work introduces a pseudo-labelling process to address the challenge of limited labelled data, expanding the training data size from 77 hours to approximately 2500 hours, leading to improved performance. Additionally, we prompt an E2E Whisper-based SGEC model with fluent transcriptions, showing a slight improvement in SGEC performance, with more significant gains in feedback generation. Finally, we assess the impact of increasing model size, revealing that while pseudo-labelled data does not yield performance gain for a larger Whisper model, training with prompts proves beneficial.

Problem

• Improving spoken grammatical error correction for language learners
• Addressing limited labeled data via pseudo-labeling expansion
• Evaluating prompting and scaling effects on model performance

Innovation

• Pseudo-labelling expands training data significantly
• Prompting Whisper model improves feedback generation
• Larger model benefits from prompts not pseudo-labels