Combining model tracing and constraint-based modeling for multistep strategy diagnoses

2025-07-18

AI Summary

Students often skip steps, merge steps, or deviate from prescribed solution strategies in multi-step problem-solving tasks, leading to inaccurate diagnostic assessments. Method: This paper proposes a novel approach integrating model tracing with constraint-based modeling. Its core innovation lies in formalizing constraints as shared semantic attributes between student inputs and predefined strategy steps—marking the first organic unification of these two paradigms. The method employs a dynamic strategy-matching algorithm coupled with educational data mining techniques to enable real-time diagnosis of nonlinear and incomplete behavioral sequences. Results: Evaluated on a quadratic equation solving dataset (n = 2,136), the system achieved fully automated diagnosis across all samples. Inter-rater agreement with two expert teachers’ manual coding of 140 samples reached perfect concordance (Cohen’s κ = 1.0), demonstrating high accuracy and robustness.

Technology Category

• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Search and Optimization: Heuristic Search

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

Model tracing and constraint-based modeling are two approaches to diagnose student input in stepwise tasks. Model tracing supports identifying consecutive problem-solving steps taken by a student, whereas constraint-based modeling supports student input diagnosis even when several steps are combined into one step. We propose an approach that merges both paradigms. By defining constraints as properties that a student input has in common with a step of a strategy, it is possible to provide a diagnosis when a student deviates from a strategy even when the student combines several steps. In this study we explore the design of a system for multistep strategy diagnoses, and evaluate these diagnoses. As a proof of concept, we generate diagnoses for an existing dataset containing steps students take when solving quadratic equations (n=2136). To compare with human diagnoses, two teachers coded a random sample of deviations (n=70) and applications of the strategy (n=70). Results show that that the system diagnosis aligned with the teacher coding in all of the 140 student steps.

Problem

• Diagnosing multistep student strategies in learning tasks
• Combining model tracing and constraint-based modeling approaches
• Validating system accuracy against human teacher diagnoses

Innovation

• Combines model tracing and constraint-based modeling
• Diagnoses multistep strategies with merged paradigms
• Validates via teacher-aligned quadratic equation diagnoses