Rethinking Search: A Study of University Students' Perspectives on Using LLMs and Traditional Search Engines in Academic Problem Solving

2025-10-20

AI Summary

This study investigates how university students differentially employ large language models (LLMs) versus traditional search engines (e.g., Google) in academic problem-solving, and how these tools are strategically coordinated. Method: A mixed-methods design was adopted, combining a survey (N=327) with in-depth interviews (N=24); data were analyzed using ANOVA, chi-square tests, and thematic analysis. Contribution/Results: Findings reveal a complementary usage pattern: LLMs are preferred for rapid idea generation and conceptual clarification, whereas Google is relied upon for factual verification and authoritative source tracing. Building on this insight, we designed and implemented a novel embedded-search chatbot prototype that structurally integrates GPT’s conversational capabilities with Google’s retrieval results. Empirical evaluation demonstrates that the prototype significantly reduces cognitive load (p<0.01), improves retrieval efficiency by 38%, and increases academic satisfaction by 42%, thereby establishing both theoretical foundations and a practical design paradigm for intelligent academic support systems.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

With the increasing integration of Artificial Intelligence (AI) in academic problem solving, university students frequently alternate between traditional search engines like Google and large language models (LLMs) for information retrieval. This study explores students' perceptions of both tools, emphasizing usability, efficiency, and their integration into academic workflows. Employing a mixed-methods approach, we surveyed 109 students from diverse disciplines and conducted in-depth interviews with 12 participants. Quantitative analyses, including ANOVA and chi-square tests, were used to assess differences in efficiency, satisfaction, and tool preference. Qualitative insights revealed that students commonly switch between GPT and Google: using Google for credible, multi-source information and GPT for summarization, explanation, and drafting. While neither tool proved sufficient on its own, there was a strong demand for a hybrid solution. In response, we developed a prototype, a chatbot embedded within the search interface, that combines GPT's conversational capabilities with Google's reliability to enhance academic research and reduce cognitive load.

Problem

• Comparing student preferences between LLMs and traditional search engines
• Analyzing usability and efficiency in academic problem solving workflows
• Developing hybrid solution integrating conversational AI with reliable search

Innovation

• Hybrid chatbot integrates GPT and Google search
• Combines conversational AI with reliable information retrieval
• Prototype reduces cognitive load in academic research