Problem
Research questions and friction points this paper is trying to address.
science problem-solving
generative AI chatbots
cognitive offloading
custom chatbot
general-purpose chatbot
Comparing the Impact of Pedagogy-Informed Custom and General-Purpose GAI Chatbots on Students' Science Problem-Solving Processes and Performance Using Heterogeneous Interaction Network Analysis
🤖 AI Summary
This study investigates the differential effects of general-purpose versus pedagogically driven, customized generative AI (GAI) chatbots on cognitive offloading and cognitive engagement during students’ scientific problem solving. Innovatively integrating Socratic questioning into the design of the customized GAI, the research employs a within-subjects counterbalanced experimental design, combining heterogeneous interaction network analysis (HINA) with dialogue log data to uncover the mechanisms through which each system shapes students’ cognitive interaction patterns. Findings reveal that students interacting with the customized bot exhibited greater interaction intensity and cognitive diversity, demonstrating a stronger tendency toward reflection rather than direct answer-seeking. Although no significant difference emerged in problem-solving quality, this work provides the first empirical evidence that embedding instructional pedagogy into GAI meaningfully influences the depth and nature of student learning processes.
📝 Abstract
Problem solving plays an essential role in science education, and generative AI (GAI) chatbots have emerged as a promising tool for supporting students' science problem solving. However, general-purpose chatbots (e.g., ChatGPT), which often provide direct, ready-made answers, may lead to students' cognitive offloading. Prior research has rarely focused on custom chatbots for facilitating students' science problem solving, nor has it examined how they differently influence problem-solving processes and performance compared to general-purpose chatbots. To address this gap, we developed a pedagogy-informed custom GAI chatbot grounded in the Socratic questioning method, which supports students by prompting them with guiding questions. This study employed a within-subjects counterbalanced design in which 48 secondary school students used both custom and general-purpose chatbot to complete two science problem-solving tasks. 3297 student-chatbot dialogues were collected and analyzed using Heterogeneous Interaction Network Analysis (HINA). The results showed that: (1) students demonstrated significantly higher interaction intensity and cognitive interaction diversity when using custom chatbot than using general-purpose chatbot; (2) students were more likely to follow custom chatbot's guidance to think and reflect, whereas they tended to request general-purpose chatbot to execute specific commands; and (3) no statistically significant difference was observed in students' problem-solving performance evaluated by solution quality between two chatbot conditions. This study provides novel theoretical insights and empirical evidence that custom chatbots are less likely to induce cognitive offloading and instead foster greater cognitive engagement compared to general-purpose chatbots. This study also offers insights into the design and integration of GAI chatbots in science education.
Innovation
Methods, ideas, or system contributions that make the work stand out.
pedagogy-informed chatbot
Socratic questioning
Heterogeneous Interaction Network Analysis
cognitive engagement
generative AI in education
🔎 Similar Papers
No similar papers found.
