Can We Delegate Learning to Automation?: A Comparative Study of LLM Chatbots, Search Engines, and Books

2024-10-02

arXiv.org

AI Summary

This study investigates whether large language model (LLM)-based chatbots in Search-as-Learning (SAL) contexts induce a trade-off between conceptual understanding and long-term retention, and how they affect learning agency. Employing a mixed-methods design—integrating surveys, behavioral log analysis, and semi-structured interviews—complemented by statistical modeling (regression and path analysis), the study compares LLMs, search engines, and textbooks across knowledge acquisition, conceptual understanding, and memory retention. Key contributions include: (1) empirical evidence that LLMs significantly enhance conceptual understanding yet yield lower knowledge retention than traditional resources; (2) no evidence of increased passive learning, but strong moderation by academic ability—high-ability learners prefer deep reading over immediate querying; and (3) academic performance positively predicts both depth of understanding and content immersion. These findings provide rigorous, educationally grounded evidence for LLM integration and inform ability-adaptive instructional design.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Learning is a key motivator behind information search behavior. With the emergence of LLM-based chatbots, students are increasingly turning to these tools as their primary resource for acquiring knowledge. However, the transition from traditional resources like textbooks and web searches raises concerns among educators. They worry that these fully-automated LLMs might lead students to delegate critical steps of search as learning. In this paper, we systematically uncover three main concerns from educators' perspectives. In response to these concerns, we conducted a mixed-methods study with 92 university students to compare three learning sources with different automation levels. Our results show that LLMs support comprehensive understanding of key concepts without promoting passive learning, though their effectiveness in knowledge retention was limited. Additionally, we found that academic performance impacted both learning outcomes and search patterns. Notably, higher-competence learners engaged more deeply with content through reading-intensive behaviors rather than relying on search activities.

Problem

• Examining perceptions and learning effects of LLM chatbots in search-as-learning contexts
• Investigating trade-offs between efficiency and depth of processing in AI-assisted learning
• Designing human-AI collaborative systems to balance learning efficiency and retention

Innovation

• Surveyed educators and students on chatbot perceptions
• Designed within-subjects experiment with three search modalities
• Used encoding-storage paradigm to assess learning effects