When Content is Goliath and Algorithm is David: The Style and Semantic Effects of Generative Search Engine

2025-09-17

AI Summary

This study investigates the source preference mechanisms of generative search engines (GSEs) when citing web content and their impact on user information diversity and retrieval efficiency. We developed a controllable simulation system based on RAG APIs and conducted a real-user randomized controlled trial. Results reveal that GSEs exhibit strong preference for sources with high language model predictability and semantic similarity. While LLM-polished content enhances overall information diversity, it induces an education-level heterogeneity effect: highly educated users complete tasks significantly faster without sacrificing diversity, whereas less-educated users achieve higher information density with stable completion time. This work provides the first empirical evidence that GSE citation bias is fundamentally semantics-driven, and elucidates its differential cognitive impacts across user groups. Our findings offer both theoretical foundations and empirical support for designing fair, interpretable, and cognitively inclusive generative search systems.

Technology Category

• Search and Optimization: Metareasoning and Metaheuristics
• Machine Learning: Learning Preferences or Rankings
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Applications of semantic technologies for improving search, browsing, recommendation, personalization

Abstract

Generative search engines (GEs) leverage large language models (LLMs) to deliver AI-generated summaries with website citations, establishing novel traffic acquisition channels while fundamentally altering the search engine optimization landscape. To investigate the distinctive characteristics of GEs, we collect data through interactions with Google's generative and conventional search platforms, compiling a dataset of approximately ten thousand websites across both channels. Our empirical analysis reveals that GEs exhibit preferences for citing content characterized by significantly higher predictability for underlying LLMs and greater semantic similarity among selected sources. Through controlled experiments utilizing retrieval augmented generation (RAG) APIs, we demonstrate that these citation preferences emerge from intrinsic LLM tendencies to favor content aligned with their generative expression patterns. Motivated by applications of LLMs to optimize website content, we conduct additional experimentation to explore how LLM-based content polishing by website proprietors alters AI summaries, finding that such polishing paradoxically enhances information diversity within AI summaries. Finally, to assess the user-end impact of LLM-induced information increases, we design a generative search engine and recruit Prolific participants to conduct a randomized controlled experiment involving an information-seeking and writing task. We find that higher-educated users exhibit minimal changes in their final outputs' information diversity but demonstrate significantly reduced task completion time when original sites undergo polishing. Conversely, lower-educated users primarily benefit through enhanced information density in their task outputs while maintaining similar completion times across experimental groups.

Problem

• Investigating citation preferences of generative search engines for predictable content
• Examining how LLM-based content polishing affects AI summary diversity
• Assessing user task performance differences across education levels with polished content

Innovation

• LLM-generated summaries with citations
• RAG APIs for citation preference analysis
• LLM-based content polishing enhances diversity