Will It Teach as Intended? How Teachers Configure Educational AI Chatbots

2026-09-24

AI Summary

This study investigates whether teachers’ pedagogical intentions are accurately translated into actual behaviors when configuring educational AI chatbots. By integrating focus groups, configuration analysis, and log-based evaluation within a generative AI experiment, the research examines how educators employ authoring tools to convert instructional goals into system settings. The findings reveal that configurable controls alone cannot guarantee pedagogical fidelity. Empirical results demonstrate high performance in chatbot responsiveness (88.9%) and persona consistency (81.5%), yet comparatively lower adherence to rules (70.4%) and goal alignment (59.3%). Consequently, this work underscores the necessity of developing intelligent authoring tools that support teachers in articulating, testing, and iteratively refining intended pedagogical behaviors.

Abstract

Teachers are increasingly using generative AI to support instruction, yet it remains unclear how pedagogical intentions are translated into chatbot configurations and reflected in chatbot behavior. We studied a teacher-facing chatbot authoring tool in professional development workshops with 27 middle school teachers, analyzing focus-group interviews alongside configuration and interaction logs. Teachers envisioned chatbots as instructional scaffolds that could provide differentiated support, extend access to assistance, and preserve student thinking within teacher-defined boundaries. Configuration analysis showed that Purpose primarily captured instructional goals and content focus, whereas Rules more often specified pedagogical behavior, guardrails, and learner-specific adaptations. Log-based evaluation showed stronger alignment for responsiveness (88.9%) and persona (81.5%) than for rules (70.4%) and purpose (59.3%). These findings show that configurable controls alone do not ensure pedagogical fidelity and highlight the need for authoring tools that help teachers express, test, and refine intended chatbot behavior.

Problem

• educational AI chatbots
• pedagogical intentions
• chatbot configuration
• pedagogical fidelity
• generative AI

Innovation

• Educational AI Chatbots
• Prompt Configuration
• Pedagogical Alignment
• Authoring Tools
• Log-based Evaluation