Warning About AI Fallibility Increases Help-Seeking in an Intelligent Tutoring System

2026-06-02

AI Summary

This study addresses the risk that students in intelligent tutoring systems may suppress help-seeking behaviors due to overreliance on AI, particularly in light of potential model hallucinations. Without altering system functionality, the authors introduce a lightweight transparency intervention—simply informing learners that the instructional agent may make errors—and provide the first empirical evidence that such a warning significantly increases students’ willingness to seek help. Through a controlled experiment and behavioral log analysis, they find that participants who received the warning made significantly more voluntary requests for hints, demonstrating that even minimal transparency can effectively modulate learner interaction strategies. Crucially, this effect emerged without compromising immediate task performance, highlighting the potential of low-cost interventions to enhance human-AI collaborative learning.

Abstract

Recent work in Technology-Enhanced Learning and Human-Computer Interaction highlights the importance of transparency and trust calibration in AI-supported learning environments as they pose a risk of hallucinations. In this study, we investigate whether a simple transparency intervention that warns students that a pedagogical agent may make mistakes affects learner behavior in a math intelligent tutoring system. We conducted a classroom experiment with 252 school students using two system versions: one including a warning message about potential system errors, and one that does not mention potential errors. Using log data, we analyzed students' problem-solving performance data, including help-seeking behavior, error rate, and time-on-task. Results show that students who were warned about potential AI errors requested significantly more hints than those in the other condition, even though the actual system behavior was exactly the same. This finding suggests that lightweight transparency interventions can influence learners' interaction strategies without necessarily improving or impairing immediate performance.

Problem

• AI fallibility
• help-seeking
• intelligent tutoring system
• trust calibration
• transparency

Innovation

• transparency intervention
• trust calibration
• help-seeking behavior
• intelligent tutoring system
• AI fallibility