Language Representations Can be What Recommenders Need: Findings and Potentials

2024-07-07

AI Summary

This work investigates whether pretrained language models (PLMs) implicitly encode user preferences and collaborative signals in their representation spaces. To this end, we propose TextCF—a purely text-driven collaborative filtering framework that eliminates the need for item ID embeddings. TextCF employs lightweight linear projections to map item title representations extracted from PLMs (e.g., LLaMA, BERT) into a recommendation-aware latent space. Theoretically, we establish a homomorphic structure between linguistic and collaborative representation spaces. Empirically, TextCF significantly outperforms state-of-the-art ID-based CF methods across multiple public benchmarks—demonstrating, for the first time, that title text alone suffices to achieve superior recommendation performance. Moreover, TextCF exhibits zero-shot recommendation capability and inherent potential for user intent awareness. Collectively, it introduces a new paradigm for recommender systems that is initialization-friendly, generalizable, and inherently interpretable.

Technology Category

Machine Learning: Learning Preferences or RankingsNatural Language Processing: Sentence-level Semantics, Textual Inference, etc.Data Mining & Knowledge Management: Recommender Systems

Application Category

User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendationSearch and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingWeb Mining and Content Analysis: Large pretrained models with web data

Abstract

Recent studies empirically indicate that language models (LMs) encode rich world knowledge beyond mere semantics, attracting significant attention across various fields. However, in the recommendation domain, it remains uncertain whether LMs implicitly encode user preference information. Contrary to prevailing understanding that LMs and traditional recommenders learn two distinct representation spaces due to the huge gap in language and behavior modeling objectives, this work re-examines such understanding and explores extracting a recommendation space directly from the language representation space. Surprisingly, our findings demonstrate that item representations, when linearly mapped from advanced LM representations, yield superior recommendation performance. This outcome suggests the possible homomorphism between the advanced language representation space and an effective item representation space for recommendation, implying that collaborative signals may be implicitly encoded within LMs. Motivated by these findings, we explore the possibility of designing advanced collaborative filtering (CF) models purely based on language representations without ID-based embeddings. To be specific, we incorporate several crucial components to build a simple yet effective model, with item titles as the input. Empirical results show that such a simple model can outperform leading ID-based CF models, which sheds light on using language representations for better recommendation. Moreover, we systematically analyze this simple model and find several key features for using advanced language representations: a good initialization for item representations, zero-shot recommendation abilities, and being aware of user intention. Our findings highlight the connection between language modeling and behavior modeling, which can inspire both natural language processing and recommender system communities.

Problem

• Explores if language models encode user preferences for recommendations
• Tests mapping language representations to item spaces for better performance
• Designs collaborative filtering models using only language representations

Innovation

• Linear mapping from LM to item representations
• Language representations replace ID-based embeddings
• Simple model with item titles outperforms ID-based CF