Interpreting Learning Under Competing Models: Joint and Stepwise Approaches for Dynamic Cognitive Diagnosis

2026-06-04

AI Summary

This study addresses the challenge of accurately tracking students’ dynamic mastery of specific skills when the Q-matrix is unknown. Building upon dynamic cognitive diagnosis models, it compares a joint estimation approach—simultaneously inferring the Q-matrix and learning trajectories—with a two-step strategy that first estimates the Q-matrix and then analyzes skill development. Leveraging reading game data and item text embeddings, the research investigates vocabulary and comprehension growth among second- to third-grade students. The authors propose a bias-corrected two-step method and use simulation studies to delineate the conditions under which each approach performs best: joint modeling proves more reliable when the Q-matrix is uncertain and items vary across grade levels. Empirical results indicate that both methods identify a general trend toward mastering both skills, yet they diverge in estimating the proportion of partial mastery in third grade, underscoring the substantive impact of modeling choices on diagnostic conclusions.

Abstract

Digital learning environments record learners' responses to individual items, making it possible to study the development of specific skills rather than overall scores. Drawing conclusions about learning from these data requires a model that links responses to latent skills and tracks how mastery changes over time. When the skills measured by each item are unknown, the analyst must decide whether to estimate this structure, the Q-matrix, jointly with the learning process, or to establish it first and study learning afterwards. We show that this decision can change substantive conclusions about how learners develop. Using dynamic cognitive diagnostic models, we analyse data from two reading games measuring vocabulary and comprehension from Grade 2 to Grade 3, with item-text embeddings providing prior information for the unknown Q-matrix. A joint analysis and a bias-corrected stepwise analysis agree that most learners move toward mastering both skills, but disagree about how many remain only partially proficient at Grade 3, changing how reading progress would be reported. A simulation study identifies when the two analyses diverge and shows that joint analysis is more reliable when the item-skill structure is uncertain and the item pool changes between grades. We provide R code for both analyses.

Problem

• cognitive diagnosis
• Q-matrix
• dynamic learning
• model comparison
• skill development

Innovation

• dynamic cognitive diagnostic models
• Q-matrix estimation
• joint vs. stepwise analysis
• item-text embeddings
• learning trajectory inference