Exploring The Interaction-Outcome Paradox: Seemingly Richer and More Self-Aware Interactions with LLMs May Not Yet Lead to Better Learning

2025-11-12

AI Summary

This study identifies and empirically validates the “Interaction–Effectiveness Paradox”: although large language models (LLMs) enable richer, metacognitively aware interactions—such as deep knowledge articulation and reflective self-monitoring—compared to search engines, they do not yield statistically significant improvements in learning outcomes. Method: A controlled experiment (N = 20) compared an LLM-based dialogue system with a conventional search interface across authentic learning tasks, integrating qualitative interaction analysis with quantitative learning assessments. Results: While LLMs enhanced interaction quality, they failed to produce a statistically significant gain in overall learning effectiveness. Contribution: This work formally defines the paradox for the first time, revealing that increased interactivity may redistribute—rather than augment—cognitive effort. It advocates for educational AI design that scaffolds, rather than supplants, learners’ active cognitive engagement, offering a novel theoretical framework and practical implications for AI-augmented learning.

Technology Category

Machine Learning: Large Multimodal Models (LMMs)Cognitive Modeling & Cognitive Systems: Social Cognition And InteractionSearch and Optimization: Learning to Search

Application Category

User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendationSemantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactionsSearch and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

While Large Language Models (LLMs) have transformed the user interface for learning, moving from keyword search to natural language dialogue, their impact on educational outcomes remains unclear. We present a controlled study (N=20) that directly compares the learning interaction and outcomes between LLM and search-based interfaces. We found that although LLMs elicit richer and nuanced interactions from a learner, they do not produce broadly better learning outcomes. In this paper, we explore this the ``Interaction-Outcome Paradox.''To explain this, we discuss the concept of a cognitive shift: the locus of student effort moves from finding and synthesizing disparate sources (search) to a more self-aware identification and articulation of their knowledge gaps and strategies to bridge those gaps (LLMs). This insight provides a new lens for evaluating educational technologies, suggesting that the future of learning tools lies not in simply enriching interaction, but in designing systems that scaffold productive cognitive work by leveraging this student expressiveness.

Problem

• Investigating why richer LLM interactions do not improve learning outcomes
• Comparing learning effectiveness between LLM dialogue and search interfaces
• Exploring how student cognitive shifts affect educational technology design

Innovation

• LLMs enable richer learner interactions
• Cognitive shift from search to self-awareness
• Scaffolding productive cognitive work through expressiveness