conduct multimodal observation

This study addresses the lack of systematic multimodal analytical approaches in synchronous online teaching. Integrating instructional orchestration theory, a multimodal interaction framework, and the data affordances of video conferencing platforms, it proposes the first reproducible, prioritized multimodal observation schema specifically designed for synchronous online classrooms. The schema encompasses observable instructor behaviors such as gestures, posture, gaze, and digital tool usage. The resulting structured observation grid not only fills a critical methodological gap in the field but also provides a practical and actionable analytical framework for future empirical research on online teaching practices.

To address subjectivity, high labor costs, and insufficient cultural adaptation in classroom observation, this study develops the first multimodal AI behavioral analysis system tailored for Asian classrooms. Methodologically, it integrates audio, video, and environmental sensor data; proposes a culturally adaptive educational AI analytics framework; designs a scoring-free, feedback-oriented instructional reflection dashboard; and establishes the first publicly available audiovisual annotation dataset of Asian classroom interactions. Key contributions include: (1) introducing interpretable, behavior-based classroom metrics; (2) achieving real-time behavioral recognition with low cognitive load and high usability—validated by eight experts from Singapore’s National Institute of Education; and (3) significantly reducing human effort required for classroom observation. The system provides teachers with objective, scalable, and context-sensitive technological support for professional development.

Traditional classroom observation faces scalability limitations due to scarce expert resources and high implementation costs, hindering widespread support for teacher professional development. This study proposes AVA-Align, an intelligent agent framework that integrates generative AI with multimodal learning to enable temporally precise analysis of long-duration classroom audio-video recordings and to automatically generate personalized, real-time feedback aligned with evidence-based teaching practices. Employing participatory design, we developed an end-to-end classroom analytics platform and validated it empirically across multiple cohorts of in-service teachers, confirming its usability and pedagogical effectiveness. Key contributions include: (1) the first deep integration of generative AI into temporal classroom understanding tasks, establishing a novel human-AI collaborative pedagogical guidance paradigm; and (2) critical theoretical reflection on privacy preservation and the boundaries of human judgment in educational AI systems.

This work addresses the scarcity of ecologically valid multimodal datasets that hinders in-depth analysis of students’ oral presentation skills and the development of automated feedback systems. To bridge this gap, we introduce the SOPHIAS dataset, collected in authentic classroom settings, comprising 50 oral presentations and subsequent Q&A sessions delivered by 65 undergraduate students. The dataset integrates eight synchronized sensor modalities—including high-definition audio-video, eye-tracking, physiological signals (via smartwatches), interaction logs (keyboard, mouse, and clicker data), and presentation slides—and is annotated with standardized evaluations from instructors, peers, and self-assessments. Spanning approximately 12 hours of recordings, SOPHIAS is publicly available on GitHub and the Science Data Bank, offering the first high-ecological-validity benchmark resource for multimodal learning analytics, automated feedback generation, and peer assessment research.

Existing surveys predominantly examine isolated components of multimodal pipelines and lack empirically grounded, pedagogically oriented integration frameworks for teaching and learning contexts. Method: This study introduces the first taxonomy and analytical framework covering five core modalities—natural language, video, sensor data, human-centered signals, and environmental logs—and proposes a novel “mid-fusion” paradigm for multimodal data integration. It further innovates by applying citation graph pruning to achieve structured, high-precision literature synthesis. Contribution/Results: Through systematic review, taxonomic modeling, and multimodal fusion design, we demonstrate that multimodal synergy enables detection of fine-grained learning behaviors imperceptible to unimodal analysis. While prediction accuracy remains largely unchanged, interpretability improves significantly, yielding deeper insights into learners’ cognitive-affective states and training outcomes.

This study addresses the lack of structured annotation benchmarks in existing classroom videos for evaluating multimodal models. The authors construct a multimodal teaching observation benchmark comprising 30 international lecture videos segmented into 5,158 fifteen-second clips, annotated with 39 binary-coded visual and non-visual dimensions. For the first time, this benchmark integrates fine-grained scene-level labels with whole-lesson expert ratings and qualitative assessments, establishing a two-tier human reference framework. They propose a Krippendorff’s alpha–based approach to construct reliability- and prevalence-aware labels and evaluate five state-of-the-art vision-language foundation models across tasks involving text-only, text-plus-image-frame, and full-lesson comprehension, using human annotations, expert scoring, and an LLM-as-judge protocol. Results reveal no single model dominates across all tasks; incorporating intermediate frames improves both true and false attribution accuracy, yet models tend to overrate instruction that is procedurally clear but lacks depth—highlighting the irreplaceable role of expert judgment in complex pedagogical assessment.

This study addresses the longstanding divide in classroom interaction research between large-scale observational approaches and in-depth ethnographic methods, which has hindered the development of an integrative framework. The authors propose a three-dimensional methodological space defined by scale, duration, and modality, and systematically examine the strengths and limitations of diverse methods regarding mechanism visibility, operationalizability, and practical translatability through comparative case studies of dialogic teaching and expert interviews. Innovatively incorporating an AI perspective, the work not only expands the boundaries of this methodological space but also offers a scalable theoretical and practical framework to guide AI-driven classroom research and tool design.

This study presents the first systematic evaluation of the generalizability of the classic classroom discourse coding scheme TalkMoves in one-on-one tutoring and multimodal interaction settings. Through expert annotation, multimodal data analysis (text, audio, and video), and Cohen’s kappa inter-rater reliability assessment, the authors compare TalkMoves with a hybrid coding scheme co-developed by AI and human annotators in terms of reliability, coverage, and multimodal applicability. Results indicate that while TalkMoves achieves higher overall inter-rater agreement (κ = 0.74), it struggles to effectively capture nonverbal cues. In contrast, the hybrid coding scheme demonstrates broader coverage and superior adaptability across modalities. These findings highlight the limitations of existing coding frameworks in tutoring contexts and provide theoretical grounding and practical guidance for designing next-generation behavioral coding frameworks tailored to multimodal tutoring interactions.

Traditional classroom observation methods suffer from high subjectivity and limited scalability, lacking objective means to assess student attentiveness. To address this gap, this study introduces BAV-Classroom, the first fine-grained video dataset of classroom behaviors collected in Vietnamese higher education institutions, annotated with nine distinct student behavior categories. The work systematically evaluates the performance of YOLO-family models for automated behavior recognition, demonstrating that YOLOv11 achieves superior accuracy and efficiency on this task. Using this model, the study reveals a significant decline in student focus during the latter segments of lectures. This research provides a reliable, data-driven tool for monitoring teaching quality and evaluating student engagement in real-world classroom settings.