🤖 AI Summary
This study addresses the challenge of uncovering deep-seated misconceptions among students regarding core difficult topics in online medical courses. To this end, it proposes a two-stage, multi-source data analysis approach that integrates quiz performance, response behaviors, and lecture transcripts. The first stage identifies high-challenge topics, while the second leverages large language models to detect cognitive biases that conventional metrics often miss. Findings are validated through expert instructor interviews and quantitative indicators. This work represents the first systematic integration of behavioral data with course content to reveal implicit misunderstandings using large language models, earning strong endorsement from domain experts. The approach successfully pinpoints several critical misconceptions, offering a scalable and practical pathway for targeted instructional interventions and efficacy evaluation.
📝 Abstract
This study presents a systematic approach to identifying and characterizing student misconceptions in online learning environments through a novel combination of quantitative performance analysis and large language model (LLM) assessment. We analyzed data from 9 course periods across 5 online biomedical science courses, encompassing 3,802 medical student enrollments. Using data from 40-50 topic-focused quizzes per course, we developed a two-stage methodology. First, we identified challenging central topics using quiz-level performance metrics. Second, we employed LLMs to characterize the underlying misconceptions in these high-priority areas. By examining student performance on first attempts across primarily multiple-choice questions (MCQs), we identified consistently challenging topics that were also central to course objectives. We then leveraged recent advances in generative AI to analyze three distinct data sources in combination: quiz question content, student response patterns, and lecture transcripts. This approach revealed actionable insights about student misconceptions that were not apparent from performance data alone. The quality of the LLM-identified misconceptions was rated as excellent by subject matter experts. We also conducted teacher interviews to assess the perceived utility of our topic identification method. Faculty found that data-driven identification of challenging topics was valuable and corroborated their own classroom observations. This methodology provides a scalable approach to characterizing student difficulties in learning environments where quizzes are used. Our findings demonstrate the potential for targeted and potentially personalized interventions in future course iterations, with clear pathways for measuring intervention effectiveness through follow-up quiz performance.