The Case Against Generation for Retrieval: Discriminative Language Models as Effective Retrievers

2026-07-28

AI Summary

This work addresses the challenges of high computational cost and factual inaccuracies when directly applying large language models (LLMs) to generative retrieval. The authors propose an LLM-native dual-tower retrieval framework that repurposes the LLM as a shared discriminative semantic encoder to jointly model user and item representations. Key innovations include EOS token pooling, knowledge distillation from a cross-encoder, cross-dataset transfer learning, and implicit reasoning within the user tower. Evaluated on three public benchmarks, the method achieves state-of-the-art or competitive performance and demonstrates significant improvements in retrieval metrics within an internal production system. Moreover, it exhibits strong robustness against model staleness and excellent scalability with increasing data volume.

Abstract

Large Language Models (LLMs) have emerged as powerful assets for recommender systems. However, deploying them as generative recommenders or zero-shot rankers at web-scale remains bottlenecked by prohibitive computational overhead and grounding challenges. In this paper, we revitalize the classic, highly efficient two-tower retrieval architecture by adapting LLMs as semantic representation backbones rather than generative engines. We introduce an LLM-native two-tower framework engineered for high-throughput, large-scale retrieval. Our architecture introduces several key innovations: a shared LLM encoder for joint user-item modeling, End-Of-Sentence (EOS) token pooling for compact sequence embedding, cross-dataset transfer learning, knowledge distillation from powerful cross-encoder teachers, and latent reasoning within the user tower. Extensive evaluation across three public benchmarks demonstrates that cross-encoder architecture outperforms current state-of-the-art (SoTA) models, while the efficient two-tower student achieves SoTA-comparable retrieval performance. Furthermore, experiments on internal large-scale production systems yield substantial topline retrieval improvements along with high resilience to model staleness and superior data scaling. Our findings demonstrate that when augmented with modern representation learning, the traditional two-tower paradigm remains an exceptionally competitive and practical solution for industrial retrieval systems.

Problem

• Large Language Models
• retrieval
• generative recommenders
• computational overhead
• grounding challenges

Innovation

• two-tower retrieval
• large language models
• knowledge distillation
• EOS token pooling
• cross-dataset transfer learning