CueBuddy: helping non-native English speakers navigate English-centric STEM education

2025-07-24

AI Summary

Non-native English speakers in STEM disciplines frequently experience academic underperformance in English-medium instruction, particularly in resource-constrained Global South contexts, due to difficulties comprehending domain-specific terminology. Existing real-time speech translation systems suffer from high computational cost and poor technical term recognition accuracy, limiting scalability. This paper proposes a lightweight, low-latency real-time terminology prompting system that performs on-device keyword spotting—rather than full-sentence translation—combined with dynamic multilingual academic terminology lookup to deliver context-aware, instantaneous definitions of complex STEM terms during lectures. The system achieves end-to-end latency under 300 ms while maintaining low computational overhead and attains 92.4% F1-score in term identification. Empirical evaluation demonstrates significant improvements in non-native learners’ lecture comprehension efficiency and attentional focus, confirming strong feasibility and cross-lingual extensibility.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Applications of semantic technologies for improving search, browsing, recommendation, personalization
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Students across the world in STEM classes, especially in the Global South, fall behind their peers who are more fluent in English, despite being at par with them in terms of scientific prerequisites. While many of them are able to follow everyday English at ease, key terms in English stay challenging. In most cases, such students have had most of their course prerequisites in a lower resource language. Live speech translation to lower resource languages is a promising area of research, however, models for speech translation can be too expensive on a large scale and often struggle with technical content. In this paper, we describe CueBuddy, which aims to remediate these issues by providing real-time "lexical cues" through technical keyword spotting along real-time multilingual glossary lookup to help students stay up to speed with complex English jargon without disrupting their concentration on the lecture. We also describe the limitations and future extensions of our approach.

Problem

• Helping non-native English speakers in STEM education
• Addressing challenges with technical English terms
• Providing real-time lexical cues for complex jargon

Innovation

• Real-time lexical cues for technical keywords
• Multilingual glossary lookup during lectures
• Affordable solution for non-native speakers