Exploring the Design Space of Real-time LLM Knowledge Support Systems: A Case Study of Jargon Explanations

2025-03-02

AI Summary

In video-based learning, domain-specific terminology often creates comprehension gaps due to learners’ heterogeneous knowledge backgrounds. To address individual cognitive constraints and contextual understanding challenges in real-time knowledge support, we designed and implemented StopGap—a prototype system that dynamically explains terminology during video playback. Through a qualitative probe study with N=24 participants, we systematically identified six key design dimensions for real-time LLM-powered knowledge support—emphasizing user autonomy, personalization, and mixed-initiative collaboration—thereby moving beyond conventional unidirectional explanation paradigms. Our approach integrates real-time LLM inference, multimodal visual representations, context-aware interaction, and human-centered qualitative evaluation. Results reveal significant inter-individual preference differences, empirically validate the efficacy of mixed-initiative mechanisms, and establish a reusable design space framework. This work provides empirical foundations for educational technology, accessibility research, and human-AI collaboration.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Knowledge gaps often arise during communication due to diverse backgrounds, knowledge bases, and vocabularies. With recent LLM developments, providing real-time knowledge support is increasingly viable, but is challenging due to shared and individual cognitive limitations (e.g., attention, memory, and comprehension) and the difficulty in understanding the user's context and internal knowledge. To address these challenges, we explore the key question of understanding how people want to receive real-time knowledge support. We built StopGap -- a prototype that provides real-time knowledge support for explaining jargon words in videos -- to conduct a design probe study (N=24) that explored multiple visual knowledge representation formats. Our study revealed individual differences in preferred representations and highlighted the importance of user agency, personalization, and mixed-initiative assistance. Based on our findings, we map out six key design dimensions for real-time LLM knowledge support systems and offer insights for future research in this space.

Problem

• Addressing knowledge gaps in communication due to diverse backgrounds and vocabularies.
• Exploring user preferences for real-time LLM knowledge support formats.
• Designing systems with user agency, personalization, and mixed-initiative assistance.

Innovation

• Real-time jargon explanation using LLMs
• Multiple visual knowledge representation formats
• User agency and personalization in assistance