A real-time metric of online engagement monitoring

2025-07-16

AI Summary

Existing online learning engagement monitoring often relies on simplistic counting heuristics or supervised predictive models requiring labeled outcomes or extensive training—limiting real-time applicability and interpretability for early intervention. To address this, we propose a course-chapter-aligned cumulative engagement metric derived directly from VLE (Virtual Learning Environment) log data, operating in an unsupervised manner without outcome labels or model training. Our approach enables real-time, fine-grained behavioral tracking while naturally conforming to the pedagogical rhythm of teaching weeks, thereby enhancing temporal alignment and educational interpretability. Validated across three undergraduate statistics courses, the metric achieves high concordance with conventional indicators from Week 3 onward; critically, by mid-semester (Weeks 6–8), it identifies all students who ultimately underperform at term-end, with predictive validity matching or exceeding baseline methods.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Video Understanding & Activity Analysis
• Reasoning under Uncertainty: Graphical Models

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Web data quality in the era of algorithmically-generated content

Abstract

Measuring online behavioural student engagement often relies on simple count indicators or retrospective, predictive methods, which present challenges for real-time application. To address these limitations, we reconceptualise an existing course-wide engagement metric to create a chapter-based version that aligns with the weekly structure of online courses. Derived directly from virtual learning environment log data, the new metric allows for cumulative, real-time tracking of student activity without requiring outcome data or model training. We evaluate the approach across three undergraduate statistics modules over two academic years, comparing it to the course-wide formulation to assess how the reconceptualisation influences what is measured. Results indicate strong alignment from as early as week 3, along with comparable or improved predictive validity for final grades in structured, lecture-based contexts. By the course midpoint, the weekly metric identifies as many low-performing students as are identifiable by the end of the course. While performance varies across modules, the chapter-based formulation offers a scalable and interpretable method for early engagement monitoring and student support.

Problem

• Develops real-time student engagement metric for online courses
• Addresses limitations of retrospective predictive methods
• Enables early identification of low-performing students

Innovation

• Chapter-based real-time engagement metric
• Uses virtual learning log data
• Scalable early student support