🤖 AI Summary
The widespread adoption of generative AI poses significant challenges to traditional academic assessment, as it often fails to authentically reflect students’ understanding and competencies. This study addresses this issue by permitting engineering students to use ChatGPT during open-book examinations while requiring submission of their interaction logs. The assessment focus thereby shifts from independent problem-solving to evaluating students’ abilities to judge, verify, and engineer prompts for AI-generated solutions. Through qualitative analysis of authentic interaction data, three distinct usage patterns emerged: answer retrieval, collaborative prompting, and critical validation. Findings indicate that students demonstrated the strongest reasoning capabilities when identifying and correcting AI errors. Moreover, transparent integration of AI not only reduced evasive behaviors but also fostered self-regulated, practice-oriented learning, offering a novel paradigm for competency-based assessment in the age of artificial intelligence.
📝 Abstract
Generative AI systems such as ChatGPT challenge traditional assumptions about academic assessment by enabling students to generate explanations, code, and solutions in real time. Rather than attempting to restrict AI use, this study investigates how students actually interact with such systems during formal evaluation. Engineering students were permitted to use ChatGPT during take-home open-book exams and were required to submit interaction transcripts alongside exam solutions. This provided direct observational evidence of reasoning processes rather than relying on self-reported behavior. Qualitative analysis revealed three progressive patterns of use: answer retrieval, guided collaboration, and critical verification. While some students initially copied questions verbatim and received generic responses, many refined prompts iteratively and tested outputs. Some of the strongest evidence of reasoning appeared when students evaluated incorrect or incomplete AI responses, revealing evaluative reasoning through debugging, comparison, and justification. The presence of generative AI shifted the cognitive task of assessment from producing solutions to assessing solution validity. The findings suggest that, in AI-mediated assessment environments, correctness of final answers alone may no longer provide sufficient evidence of comprehension. Instead, competencies such as prompt formulation, verification, and judgment become visible indicators of learning. Transparent integration of AI appeared to reduce focus on rule avoidance and promote self-regulation. Assessments should evolve to evaluate reasoning about solutions rather than independent solution production. Generative AI therefore does not invalidate assessment but has the potential to expose deeper forms of understanding aligned with professional practice.