Augment or Not? A Comparative Study of Pure and Augmented Large Language Model Recommenders

2025-05-29

AI Summary

This paper addresses the lack of systematic taxonomy and fair evaluation in LLM-based recommender systems. We propose the first structured classification framework—distinguishing *pure LLM-based* from *enhanced LLM-based* approaches—and introduce RecEval, an open-source, reproducible benchmark comprising 12 datasets and 8 baseline methods. Methodologically, we integrate semantic understanding, prompt engineering, and external modules (e.g., collaborative filtering, knowledge graphs), conducting controlled-variable experiments under standardized protocols. Results show that pure LLM recommenders excel in cold-start scenarios and interpretability, whereas enhanced variants significantly outperform in long-tail distributions and sparse interaction settings. Our key contributions are: (1) a theory-grounded classification framework; (2) the first open-source, unified evaluation platform for LLM-based recommendation; and (3) empirically grounded guidelines delineating the applicability boundaries and design trade-offs between the two paradigms.

Technology Category

• Data Mining & Knowledge Management: Recommender Systems
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large language models (LLMs) have introduced new paradigms for recommender systems by enabling richer semantic understanding and incorporating implicit world knowledge. In this study, we propose a systematic taxonomy that classifies existing approaches into two categories: (1) Pure LLM Recommenders, which rely solely on LLMs, and (2) Augmented LLM Recommenders, which integrate additional non-LLM techniques to enhance performance. This taxonomy provides a novel lens through which to examine the evolving landscape of LLM-based recommendation. To support fair comparison, we introduce a unified evaluation platform that benchmarks representative models under consistent experimental settings, highlighting key design choices that impact effectiveness. We conclude by discussing open challenges and outlining promising directions for future research. This work offers both a comprehensive overview and practical guidance for advancing next-generation LLM-powered recommender.

Problem

• Comparing pure and augmented LLM-based recommender systems
• Developing a taxonomy for LLM-based recommendation approaches
• Evaluating performance of LLM recommenders with unified benchmarks

Innovation

• Systematic taxonomy for LLM recommenders
• Unified evaluation platform for benchmarking
• Integration of non-LLM techniques