Evolving Paradigms in Task-Based Search and Learning: A Comparative Analysis of Traditional Search Engine with LLM-Enhanced Conversational Search System

2025-11-28

AI Summary

Traditional keyword-based search engines lack support for multi-step reasoning and exploratory learning, while LLM-powered conversational search (e.g., ChatGPT) may fundamentally reshape query formulation, information navigation, and knowledge construction—yet empirical evidence of its impact remains scarce. Method: This study conducts the first systematic comparison between traditional and LLM-augmented conversational search across task-oriented and learning-oriented search tasks, integrating search behavior log analysis, interactive dialogue interfaces, and multidimensional cognitive assessments (conceptual understanding depth, knowledge integration, and critical thinking). Contribution/Results: LLMs significantly improve query formulation efficiency and information integration quality, fostering deeper conceptual understanding; however, they also induce overreliance on generated content and diminish users’ verification awareness. The findings reveal generative AI’s structural influence on the cognitive mechanisms underlying information retrieval, providing both theoretical grounding and empirical support for designing human-AI collaborative, learning-centered search systems.

Technology Category

Data Mining & Knowledge Management: Conversational Systems for Recommendation & RetrievalSearch and Optimization: Learning to SearchMachine Learning: Large Multimodal Models (LMMs)

Application Category

Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved informationSemantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMsUser Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Language Models (LLMs) are rapidly reshaping information retrieval by enabling interactive, generative, and inference-driven search. While traditional keyword-based search remains central to web and academic information access, it often struggles to support multi-step reasoning and exploratory learning tasks. LLM-powered search interfaces, such as ChatGPT and Claude, introduce new capabilities that may influence how users formulate queries, navigate information, and construct knowledge. However, empirical understanding of these effects is still limited. This study compares search behavior and learning outcomes in two environments: a standard search engine and an LLM-powered search system. We investigate (1) how search strategies, query formulation, and evaluation behaviors differ across systems, and (2) how LLM use affects comprehension, knowledge integration, and critical thinking during search-based learning tasks. Findings offer insight into how generative AI shapes information-seeking processes and contribute to ongoing discussions in information retrieval, human-AI interaction, and technology-supported learning.

Problem

• Compares search behavior in traditional vs LLM-enhanced systems
• Examines how LLM use affects learning outcomes and critical thinking
• Investigates differences in query formulation and information evaluation strategies

Innovation

• LLM-enhanced conversational search system
• Comparative analysis of search behavior
• Generative AI for information-seeking processes