AI-Generated Traces for Novice Programmers: Learning Effects and Learner Differences in a Multi-Institutional Study

2026-06-02

AI Summary

This study addresses the persistent challenge novice learners face in understanding program execution dynamics, a gap inadequately addressed by existing visualization tools and compounded by the lack of empirical evaluation of AI-generated instructional content. The authors propose Generative Animated Traces (GATs), an approach that leverages artificial intelligence to automatically produce narrative animations integrating source code, runtime states, and conceptual analogies to support CS1 students’ comprehension of program behavior. In the first large-scale, multi-institutional experiment of its kind, the research evaluates analogy-driven, AI-generated animations within both Python and Java introductory programming courses, combining educational experimentation with learning analytics. Findings indicate that GATs can selectively enhance immediate learning outcomes; however, their efficacy is context-dependent, exhibits short-term effects, and is significantly moderated by learners’ engagement patterns, underscoring the critical need for personalized instructional interventions.

Abstract

Introductory programming (CS1) courses often struggle to support students' understanding of program execution. While visualizations can make execution processes explicit, their effectiveness depends on design and context, and empirical evidence for AI-generated visualizations remains limited. We propose Generated Animated Traces (GATs), AI-generated, analogy-based, narrated animations that coordinate source code, execution state, and conceptual analogies. We conduct a study at two institutions in CS1 courses (Python, N=961; Java N=151) comparing GATs to textual explanations. We measure immediate learning performance and experience, end-of-course engagement and exam performance. Results show that GATs can yield selective benefits for immediate learning, but benefits are context-dependent and short-term. We observe that GATs' influence on performance is moderated by learner engagement profiles. This finding underscores the importance of personalized approaches.

Problem

• program execution understanding
• novice programmers
• learning effects
• learner differences
• AI-generated visualizations

Innovation

• Generated Animated Traces
• AI-generated visualizations
• program execution understanding
• learner engagement profiles
• personalized learning