A Case Study Reexamining the Cold-Start Problem in Knowledge Tracing Models and Implications for SafeInsights, an Education Research Infrastructure

2026-06-09

AI Summary

This study addresses the challenges of cold-start and limited generalization faced by knowledge tracing models when applied to new datasets and diverse item types. Building upon and extending the work of Zhang et al. (2021), we conduct the first systematic evaluation of mainstream knowledge tracing models on the FoundationalASSIST dataset, assessing their performance stability and reproducibility across varying practice opportunities and question types. Furthermore, we validate the capability of SafeInsights—a privacy-preserving research infrastructure—to support educational data mining. Our findings demonstrate that model performance is significantly influenced by both student practice trajectories and item characteristics, while also confirming that SafeInsights effectively enables privacy-safe, reproducible research in educational data science.

Abstract

Knowledge tracing (KT) models are widely used to predict students' evolving knowledge states from their learning history. However, many KT models are evaluated using specific datasets, platforms, and learning contexts, raising questions about whether reported model performance replicates and generalizes across newer datasets that vary in context. This paper replicates and extends Zhang et al. (2021), which examined the cold-start problem in KT models and found that deep-learning-based KT models performed better, partly because of stronger predictions when students began practicing a skill. Using a more recent ASSISTments dataset, FoundationalASSIST, we replicate the previous analysis by evaluating model performance across opportunities to practice and extend the analysis by examining performance across problem types, including fill-in-the-blank, multiple-choice select-one, multiple-choice select-all, and order/sort problems. Results show that KT model performance varies across both student practice trajectories and problem types. Beyond the empirical replication, this study identifies practical challenges in reproducing educational data mining studies and serves as a proof of concept, showing how privacy-preserving research infrastructures such as SafeInsights can be leveraged to facilitate educational research and support replication analyses.

Problem

• knowledge tracing
• cold-start problem
• replicability
• generalization
• educational data mining

Innovation

• knowledge tracing
• cold-start problem
• replication study
• SafeInsights
• problem-type generalization