AI Meets the Classroom: When Do Large Language Models Harm Learning?

2024-08-29

AI Summary

This study investigates when large language models (LLMs) enhance or hinder learning in educational contexts, focusing on usage mode (substitutive vs. supplementary), learning breadth versus depth, and the dynamic role of prior knowledge gaps. Employing a pre-registered controlled experiment with incentivized behavioral measures and a mixed-methods design (lab + field), it integrates learning logs and validated comprehension assessments to empirically disentangle the bidirectional effects of these two usage modes for the first time. Results show no overall learning effect of LLMs; substitutive use increases topical coverage but significantly impairs conceptual depth, whereas supplementary use fails to expand breadth yet markedly improves comprehension. Critically, LLMs widen the achievement gap between high- and low-prior-knowledge learners. The study proposes a “context–need” alignment framework for educational LLM deployment, offering both theoretical grounding and actionable guidelines for evidence-based LLM integration in learning environments.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

The effect of large language models (LLMs) in education is debated: Previous research shows that LLMs can help as well as hurt learning. In two pre-registered and incentivized laboratory experiments, we find no effect of LLMs on overall learning outcomes. In exploratory analyses and a field study, we provide evidence that the effect of LLMs on learning outcomes depends on usage behavior. Students who substitute some of their learning activities with LLMs (e.g., by generating solutions to exercises) increase the volume of topics they can learn about but decrease their understanding of each topic. Students who complement their learning activities with LLMs (e.g., by asking for explanations) do not increase topic volume but do increase their understanding. We also observe that LLMs widen the gap between students with low and high prior knowledge. While LLMs show great potential to improve learning, their use must be tailored to the educational context and students' needs.

Problem

• Impact of LLMs on learning outcomes in education
• Behavioral effects of LLM usage on topic volume and understanding
• LLMs widen knowledge gap between low and high prior knowledge students

Innovation

• LLMs used in laboratory experiments
• LLMs impact varies with usage behavior
• Tailored LLM use enhances learning outcomes