DeBERTa-KC: A Transformer-Based Classifier for Knowledge Construction in Online Learning Discourse

2025-10-21

AI Summary

This study addresses the high cost and low efficiency of manual annotation for knowledge construction (KC) levels in online scientific learning discussions. We propose an end-to-end automatic classification method based on DeBERTa-v3 to fine-grainedly identify four KC discourse types: non-KC, sharing, exploring, and negotiating. To tackle severe class imbalance and enhance discriminative sensitivity—particularly for higher-order cognitive discourses—we innovatively integrate Focal Loss, label smoothing, and R-Drop regularization. Under ten-fold cross-validation, our model achieves a macro-F1 score of 0.836, significantly outperforming conventional and state-of-the-art baseline models. Results demonstrate both effectiveness and reproducibility in informal digital learning contexts, offering a scalable solution for automated KC discourse analysis.

Technology Category

• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification
• Natural Language Processing: Discourse, Pragmatics & Argument Mining
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

This study presents DeBERTa-KC, a transformer-based model for automatic classification of knowledge construction (KC) levels in online science learning discourse. Using comments collected from four popular YouTube science channels (2022--2024), a balanced corpus of 20,000 manually annotated samples was created across four KC categories: extit{nonKC}, extit{Share}, extit{Explore}, and extit{Negotiate}. The proposed model extends DeBERTa-v3 with Focal Loss, Label Smoothing, and R-Drop regularization to address class imbalance and enhance generalization. A reproducible end-to-end pipeline was implemented, encompassing data extraction, annotation, preprocessing, training, and evaluation. Across 10-fold stratified cross-validation, DeBERTa-KC achieved a macro-F1 of $0.836 pm 0.008$, significantly out-performing both classical and transformer baselines ($p<0.01$). Per-category results indicate strong sensitivity to higher-order epistemic engagement, particularly in extit{Explore} and extit{Negotiate} discourse. These findings demonstrate that large language models can effectively capture nuanced indicators of knowledge construction in informal digital learning environments, offering scalable, theory-informed approaches to discourse analysis and the development of automated tools for assessing epistemic engagement.

Problem

• Classifying knowledge construction levels in online learning discourse
• Addressing class imbalance in educational data classification
• Automating epistemic engagement assessment in digital learning environments

Innovation

• Transformer model with Focal Loss for class imbalance
• Label Smoothing and R-Drop for enhanced generalization
• End-to-end pipeline for reproducible discourse classification