The Conversational Exam: A Scalable Assessment Design for the AI Era

2026-01-15

AI Summary

Traditional assessment methods are prone to failure when generative AI tools are readily accessible, often leading to misjudgments of student competence and a disconnect between learning and evaluation. This work proposes a dialogic examination model grounded in human-AI interaction principles, which integrates live group coding defenses, controlled access to AI assistance, and real-time instructor supervision within an authentic development environment to assess students’ genuine capabilities. The approach uniquely balances authenticity, cheating resistance, and scalability—demonstrated by its successful deployment in evaluating 58 students over two days—thereby confirming its feasibility for routine instructional settings. Furthermore, the study provides educators with a reusable implementation framework to adopt this assessment paradigm in their own contexts.

Technology Category

• Humans and AI: Interaction Techniques and Devices
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Machine Learning: Evaluation and Analysis

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Traditional assessment methods collapse when students use generative AI to complete work without genuine engagement, creating an illusion of competence where they believe they're learning but aren't. This paper presents the conversational exam -- a scalable oral examination format that restores assessment validity by having students code live while explaining their reasoning. Drawing on human-computer interaction principles, we examined 58 students in small groups across just two days, demonstrating that oral exams can scale to typical class sizes. The format combines authentic practice (students work with documentation and supervised AI access) with inherent validity (real-time performance cannot be faked). We provide detailed implementation guidance to help instructors adapt this approach, offering a practical path forward when many educators feel paralyzed between banning AI entirely or accepting that valid assessment is impossible.

Problem

• assessment validity
• generative AI
• conversational exam
• authentic learning
• oral examination

Innovation

• conversational exam
• scalable assessment
• generative AI
• oral examination
• authentic assessment