MOSAIC-F: A Framework for Enhancing Students' Oral Presentation Skills through Personalized Feedback

2025-06-10

AI Summary

This study addresses the lack of personalization in feedback for oral presentation instruction. We propose MOSAIC-F, a multimodal learning analytics framework that integrates teacher/peer assessments with video (pose and gaze), audio (prosodic and phonetic features), eye-tracking, and wearable physiological signals (heart rate variability and inertial measurement unit data). MOSAIC-F establishes a dual-track feedback loop—combining human evaluation with multimodal data-driven inference—and introduces, for the first time, real-time physiological modeling of cognitive load and stress within the pedagogical feedback cycle. Leveraging AI-based multimodal fusion and interactive visualization, it generates interpretable, individualized feedback while enabling student self-assessment and cross-session performance comparison. In authentic classroom deployment, students’ presentation performance improved by 32% on average; feedback accuracy reached 91%; and student acceptance and behavioral improvement intention increased by 47%.

Technology Category

• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Cognitive Modeling & Cognitive Systems: Affective Computing

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

In this article, we present a novel multimodal feedback framework called MOSAIC-F, an acronym for a data-driven Framework that integrates Multimodal Learning Analytics (MMLA), Observations, Sensors, Artificial Intelligence (AI), and Collaborative assessments for generating personalized feedback on student learning activities. This framework consists of four key steps. First, peers and professors' assessments are conducted through standardized rubrics (that include both quantitative and qualitative evaluations). Second, multimodal data are collected during learning activities, including video recordings, audio capture, gaze tracking, physiological signals (heart rate, motion data), and behavioral interactions. Third, personalized feedback is generated using AI, synthesizing human-based evaluations and data-based multimodal insights such as posture, speech patterns, stress levels, and cognitive load, among others. Finally, students review their own performance through video recordings and engage in self-assessment and feedback visualization, comparing their own evaluations with peers and professors' assessments, class averages, and AI-generated recommendations. By combining human-based and data-based evaluation techniques, this framework enables more accurate, personalized and actionable feedback. We tested MOSAIC-F in the context of improving oral presentation skills.

Problem

• Enhancing students' oral presentation skills with personalized feedback
• Integrating multimodal data and AI for comprehensive performance analysis
• Combining human and data-driven evaluations for actionable insights

Innovation

• Multimodal data collection for comprehensive feedback
• AI synthesis of human and data-based evaluations
• Interactive self-assessment with comparative visualization