Students' Feedback Requests and Interactions with the SCRIPT Chatbot: Do They Get What They Ask For?

2025-07-23

AI Summary

This study investigates the alignment between learners’ feedback requests and AI-generated responses in generative AI–powered programming education, focusing on the trade-off between instructional guidance and interactional flexibility. Method: We designed and implemented SCRIPT, an educational chatbot built on ChatGPT-4o-mini, integrating preset prompt–driven structured scaffolding with open-ended dialogue capabilities, and introduced a feedback-type matching mechanism. Using empirical interaction data from 136 students completing programming tasks, we conducted sequential pattern analysis and response accuracy evaluation. Contribution/Results: We first identify statistically significant sequential patterns in learners’ feedback requests, exposing the challenge of dynamically adapting to evolving learning intentions. Experimental results show that 75% of bot responses correctly match the requested feedback type while strictly adhering to prompt constraints. These findings establish a novel design paradigm—grounded in empirical evidence—for enhancing explainability, controllability, and adaptivity in generative AI–enabled educational tools.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Humans and AI: Game Design — Procedural Content Generation & Storytelling
• Intelligent Robots: Embodied AI

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

Building on prior research on Generative AI (GenAI) and related tools for programming education, we developed SCRIPT, a chatbot based on ChatGPT-4o-mini, to support novice learners. SCRIPT allows for open-ended interactions and structured guidance through predefined prompts. We evaluated the tool via an experiment with 136 students from an introductory programming course at a large German university and analyzed how students interacted with SCRIPT while solving programming tasks with a focus on their feedback preferences. The results reveal that students' feedback requests seem to follow a specific sequence. Moreover, the chatbot responses aligned well with students' requested feedback types (in 75%), and it adhered to the system prompt constraints. These insights inform the design of GenAI-based learning support systems and highlight challenges in balancing guidance and flexibility in AI-assisted tools.

Problem

• Evaluating student feedback preferences with SCRIPT chatbot
• Assessing alignment of chatbot responses to feedback requests
• Balancing guidance and flexibility in AI learning tools

Innovation

• ChatGPT-4o-mini based chatbot for programming education
• Open-ended interactions with structured guidance prompts
• Analyzed student feedback preferences and interaction sequences