Innovation
Methods, ideas, or system contributions that make the work stand out.
Constraint-First Design
Generative AI Teaching Partner
Interaction Records
Self-Report Limits
AI-Assisted Learning
Fluency Without Evidence: Constraint-First Design and the Limits of Self-Report in AI-Assisted Learning
🤖 AI Summary
This study addresses the lack of authentic curricular validation regarding AI teaching companions that scaffold reasoning rather than provide direct answers, alongside the overreliance on self-reported data in existing research. Employing a design-based research methodology within a graduate course, this work implements a “constraint-first” generative AI pedagogical strategy and utilizes mixed methods to comparatively analyze student self-report questionnaires against interaction logs. The findings reveal significant improvements in AI literacy but no measurable change in critical thinking. Notably, self-reported data overestimated competency gains, whereas interaction logs exposed uneven strategy implementation. By innovatively proposing a constraint-first sequential design, this project demonstrates the necessity of substituting behavioral data for self-reports, thereby offering a novel paradigm for evaluating AI-integrated instruction.
📝 Abstract
A generative AI teaching partner should support reasoning over supplying conclusions; however, this has not been tested against learning in an authentic course. Drawing on design-based research, we specify the position as a conjecture map and report a first design cycle in two graduate-level research methods courses. Students used an AI teaching partner employing a constraint-first sequence requiring them to state and justify positions before receiving questions. Pre- and post-measures of AI literacy, critical thinking, and metacognitive awareness were collected alongside interaction records. AI literacy increased, concentrating in understanding AI, whereas critical thinking, awareness, and knowledge did not change. Since changes were limited to self-report measures, they may reflect growth in confidence instead of capacity. Interaction records, meanwhile, showed brief exchanges, uneven enactment of the constraint-first sequence, and missing records. These findings show why AI-supported learning requires interaction records to provide a more defensible basis for AI-supported designs than self-reports.
🔎 Similar Papers
No similar papers found.
