Exploring undercurrents of learning tensions in an LLM-enhanced landscape: A student-centered qualitative perspective on LLM vs Search

2025-04-03

AI Summary

This study investigates the differential cognitive functions and educational value of large language models (LLMs; e.g., ChatGPT) versus search engines (e.g., Google) in students’ self-directed learning of novel topics. Employing a within-subjects balanced experimental design, complemented by semi-structured interviews and thematic analysis, it systematically compares student-centered usage preferences, epistemic decision-making processes, and learning tensions arising during information discovery, synthesis, and verification. The study identifies, for the first time, a dynamic trade-off between “cognitive effort reduction” and “epistemic agency”—the learner’s capacity to critically evaluate, justify, and regulate knowledge claims. Six prototypical learning tension scenarios are empirically derived. Building on these findings, the research proposes a contextualized technology adoption framework that informs evidence-based design of educational technologies, pedagogical integration of LLMs, and cultivation of critical digital literacy.

Technology Category

Machine Learning: Large Multimodal Models (LMMs)Search and Optimization: Learning to SearchCognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved informationUser Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendationSemantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large language models (LLMs) are transforming how students learn by providing readily available tools that can quickly augment or complete various learning activities with non-trivial performance. Similar paradigm shifts have occurred in the past with the introduction of search engines and Wikipedia, which replaced or supplemented traditional information sources such as libraries and books. This study investigates the potential for LLMs to represent the next shift in learning, focusing on their role in information discovery and synthesis compared to existing technologies, such as search engines. Using a within-subjects, counterbalanced design, participants learned new topics using a search engine (Google) and an LLM (ChatGPT). Post-task follow-up interviews explored students' reflections, preferences, pain points, and overall perceptions. We present analysis of their responses that show nuanced insights into when, why, and how students prefer LLMs over search engines, offering implications for educators, policymakers, and technology developers navigating the evolving educational landscape.

Problem

• Compare LLMs and search engines for student learning
• Explore student preferences between ChatGPT and Google
• Analyze educational impact of LLMs versus traditional tools

Innovation

• LLM vs Search comparative study
• Student-centered qualitative analysis
• Within-subjects counterbalanced design