Towards AI Agents for Course Instruction in Higher Education: Early Experiences from the Field

2025-10-23

AI Summary

Scaling high-quality, reflective learning in graduate cloud computing courses remains challenging due to instructor workload and pedagogical constraints. Method: We propose a “Conversational AI–Human Instructor” hybrid teaching framework, deploying explainable, workflow-integrated pedagogical agents powered by large language models (LLMs) to enable real-time, inquiry-based interaction within course activities. We further design a multidimensional engagement analytics framework—measuring thematic breadth, inquiry depth, and turn-level elaboration—to quantitatively assess learning processes from dialogue logs. Contribution/Results: Two iterative teaching experiments demonstrate that students increasingly initiate clarification queries and deep, follow-up questions, with participation patterns evolving significantly across instructional phases. The results empirically validate the feasibility and efficacy of structured conversational AI in supporting high-fidelity, replicable, and scalable instruction in higher education.

Technology Category

• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

This article presents early findings from designing, deploying and evaluating an AI-based educational agent deployed as the primary instructor in a graduate-level Cloud Computing course at IISc. We detail the design of a Large Language Model (LLM)-driven Instructor Agent, and introduce a pedagogical framework that integrates the Instructor Agent into the course workflow for actively interacting with the students for content delivery, supplemented by the human instructor to offer the course structure and undertake question--answer sessions. We also propose an analytical framework that evaluates the Agent--Student interaction transcripts using interpretable engagement metrics of topic coverage, topic depth and turn-level elaboration. We report early experiences on how students interact with the Agent to explore concepts, clarify doubts and sustain inquiry-driven dialogue during live classroom sessions. We also report preliminary analysis on our evaluation metrics applied across two successive instructional modules that reveals patterns of engagement evolution, transitioning from broad conceptual exploration to deeper, focused inquiry. These demonstrate how structured integration of conversational AI agents can foster reflective learning, offer a reproducible methodology for studying engagement in authentic classroom settings, and support scalable, high-quality higher education.

Problem

• Developing AI instructor agents for higher education courses
• Evaluating student engagement through interpretable interaction metrics
• Integrating conversational AI with human instructors for scalable education

Innovation

• LLM-driven Instructor Agent for course delivery
• Pedagogical framework integrating AI with human oversight
• Analytical framework evaluating engagement via interpretable metrics