Do intelligent tutoring systems benefit K-12 students? A meta-analysis and evaluation of heterogeneity of treatment effects in the U.S

2025-11-07

AI Summary

This study examines the impact of Intelligent Tutoring Systems (ITS) on academic achievement among U.S. K–12 students and identifies boundary conditions through meta-analysis. Synthesizing 18 studies yielding 77 effect sizes, it uniquely integrates MetaForest—a machine learning-based moderator analysis method—with conventional meta-analytic techniques to detect critical moderators, including intervention duration, instructional design features, and example provision. Results indicate a statistically significant overall positive effect of ITS on learning outcomes (Hedges’ *g* = 0.271). The effect is robust across elementary and secondary grades but attenuated in rural schools. Shorter interventions (≤12 weeks) and instructional designs embedding worked examples yield stronger effects. The findings elucidate key contextual factors governing ITS efficacy, offering empirical guidance for precision implementation and differentiated deployment of educational technology in diverse school settings.

Technology Category

• Intelligent Robots: Multi-Robot Systems
• Multiagent Systems: Multiagent Learning
• Machine Learning: Neuro-Symbolic Learning

Application Category

• User Modeling, Personalization and Recommendation: Studies of user behavior, including longitudinal effects of personalized systems
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

To expand the use of intelligent tutoring systems (ITS) in K-12 schools, it is essential to understand the conditions under which their use is most beneficial. This meta-analysis evaluated the heterogeneity of ITS effects across studies focusing on elementary, middle, and high schools in the U.S. It included 18 studies with 77 effect sizes across 11 ITS. Overall, there was a significant positive effect size of ITS on U.S. K-12 students'learning outcomes (g=0.271, SE=0.011, p=0.001). Furthermore, effect sizes were similar across elementary and middle schools, and for low-achieving students, but were lower in studies including rural schools. A MetaForest analysis showed that providing worked-out examples, intervention duration, intervention condition, type of learning outcome, and immediate measurement were the most important moderators of treatment effects.

Problem

• Evaluating intelligent tutoring system effectiveness for K-12 students
• Analyzing heterogeneity of treatment effects across US schools
• Identifying key moderators influencing tutoring system outcomes

Innovation

• Meta-analysis evaluated intelligent tutoring systems effects
• Identified key moderators like worked examples duration
• Assessed heterogeneity across school types and students