Dude, where's my utterance? Evaluating the effects of automatic segmentation and transcription on CPS detection

2025-07-06

AI Summary

Automated analysis of collaborative problem-solving (CPS) behaviors in classroom settings remains challenging due to reliance on labor-intensive manual transcription and annotation. Method: We systematically evaluate the impact of automatic speech segmentation and ASR-based transcription—applied to the public Weights Task Dataset—on CPS behavior detection, comparing performance against gold-standard human annotations. Contribution/Results: The end-to-end automated pipeline achieves comparable F1 scores to manual transcription (p > 0.05), while reducing utterance count by 26.5%, confirming its practical viability for scalable classroom interaction analysis. However, coarser segmentation granularity incurs non-negligible information loss, highlighting a trade-off between processing efficiency and analytical depth. This work constitutes the first systematic assessment of how full-stack speech processing pipelines affect CPS recognition accuracy, providing both methodological guidance and empirical evidence for low-intervention, high-throughput collaborative learning analytics in educational contexts.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Evaluation and Analysis
• Cognitive Modeling & Cognitive Systems: Affective Computing

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Web Mining and Content Analysis: Normalization, clustering, classification, and summarization of Web text

Abstract

Collaborative Problem-Solving (CPS) markers capture key aspects of effective teamwork, such as staying on task, avoiding interruptions, and generating constructive ideas. An AI system that reliably detects these markers could help teachers identify when a group is struggling or demonstrating productive collaboration. Such a system requires an automated pipeline composed of multiple components. In this work, we evaluate how CPS detection is impacted by automating two critical components: transcription and speech segmentation. On the public Weights Task Dataset (WTD), we find CPS detection performance with automated transcription and segmentation methods is comparable to human-segmented and manually transcribed data; however, we find the automated segmentation methods reduces the number of utterances by 26.5%, impacting the the granularity of the data. We discuss the implications for developing AI-driven tools that support collaborative learning in classrooms.

Problem

• Evaluating impact of automated transcription on CPS detection
• Assessing effect of automatic segmentation on utterance granularity
• Developing AI tools for classroom collaborative learning support

Innovation

• Automated transcription for CPS detection
• Automated speech segmentation analysis
• Evaluated impact on utterance granularity