VCEval: Rethinking What is a Good Educational Video and How to Automatically Evaluate It

2024-06-15

arXiv.org

AI Summary

To address the inconsistent quality of online educational videos and the lack of automated assessment tools, this paper introduces the first large-scale multimodal dataset of instructional videos paired with corresponding teaching materials. We systematically formulate three foundational principles for high-quality educational videos—pedagogical soundness, structural coherence, and expressive clarity—and innovatively cast video quality assessment as a structured multiple-choice question-answering task. Our method integrates pedagogy-aware prompt engineering, language-model-driven fine-grained scoring, and principle-based interpretability analysis. Evaluated on our newly constructed benchmark, our approach achieves a 23.5% accuracy improvement over baseline methods and effectively discriminates between high- and low-quality courses. Crucially, scores are fully attributable and traceable to specific pedagogical principles, enabling actionable, instructor-targeted feedback and data-informed course refinement.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Machine Learning: Multimodal Learning
• Knowledge Representation and Reasoning: Qualitative Reasoning

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Web data quality in the era of algorithmically-generated content

Abstract

Online courses have significantly lowered the barrier to accessing education, yet the varying content quality of these videos poses challenges. In this work, we focus on the task of automatically evaluating the quality of video course content. We have constructed a dataset with a substantial collection of video courses and teaching materials. We propose three evaluation principles and design a new evaluation framework, extit{VCEval}, based on these principles. The task is modeled as a multiple-choice question-answering task, with a language model serving as the evaluator. Our method effectively distinguishes video courses of different content quality and produces a range of interpretable results.

Problem

• Online Education
• Video Quality Assessment
• Automated Evaluation

Innovation

• VCEval
• Automatic Quality Evaluation
• Language Understanding Model