Towards Breaking the Learning System Wall Using Multimodal Tutoring Transcriptions

2026-09-28

AI Summary

This study addresses the generalization bottleneck in cross-platform learning modeling caused by data heterogeneity, commonly referred to as the "learning system wall." To overcome this challenge, we propose an AI-driven transcription framework based on multimodal fusion. By integrating computer vision, speech recognition, and natural language processing, the framework transforms unstructured screen recordings into unified script-style transcripts with precise temporal alignment between dialogue flows and system logs. This approach transcends the limitations of single-source data, enabling automated conversion from raw video to structured transcriptions. Our preliminary results validate the feasibility of uniformly capturing and analyzing learning processes across heterogeneous systems, thereby offering a novel paradigm for developing generalizable cross-platform learning models.

Abstract

Past research using log data has faced the "learning system wall," whereby few methods exist for generalizing models of student learning across platforms. Increasingly, online learning is captured by richer forms of data, including dialog and video, with new affordances. An example of this is remote tutoring programs, where human tutors support students who use learning systems while video conferencing. Toward better platform-general modeling of learning, we introduce an AI-driven multimodal transcription system that processes screen-recording videos into unified screenplay-style transcripts containing audio dialogue and annotated learning log actions. We describe a planned method for temporally aligning AI-generated multimodal transcripts with MATHia learning logs and for identifying and classifying student learning processes to align with MATHia logs. Lastly, we highlight challenges and potential solutions in capturing learning processes in one system, offering initial steps towards generalizing log data across diverse systems.

Problem

• learning system wall
• cross-platform generalization
• multimodal tutoring
• log data
• student learning processes

Innovation

• Multimodal Transcription
• Cross-platform Generalization
• Learning System Wall
• Temporal Alignment
• AI-driven