Retrieval Augmented Conversational Recommendation with Reinforcement Learning

2026-04-06

AI Summary

This work addresses the limitations of existing large language model (LLM)-driven conversational recommender systems, which rely on static pre-trained knowledge, struggle with novel items, and lack a unified retrieval-augmented mechanism. To overcome these challenges, the authors propose RAR, a novel framework that dynamically bridges retrieval and generation in conversational recommendation for the first time. RAR operates in two stages: it first retrieves candidate items based on user history and then generates recommendations by fusing dialogue context with retrieved results. Additionally, it incorporates a reinforcement learning mechanism guided by LLM feedback to optimize the retriever. A large-scale structured movie corpus is constructed to support this approach. Experimental results demonstrate that RAR significantly outperforms state-of-the-art methods across multiple benchmarks, effectively reducing hallucination while enhancing recommendation accuracy, factuality, and contextual awareness.

Abstract

Large language models (LLMs) exhibit enhanced capabilities in language understanding and generation. By utilizing their embedded knowledge, LLMs are increasingly used as conversational recommender systems (CRS), achieving improved performance across diverse scenarios. However, existing LLM-based methods rely on pretrained knowledge without external retrieval mechanisms for novel items. Additionally, the lack of a unified corpus poses challenges for integrating retrieval augmentation into CRS. Motivated by these challenges, we present RAR, a novel two-stage retrieval augmented conversational recommendation framework that aligns retrieval and generation to enhance both performance and factuality. To support this framework and provide a unified corpus, we construct a large-scale movie corpus, comprising over 300k movies with rich metadata, such as titles, casts and plot summaries. Leveraging this data, our primary contribution is RAR, the first framework to departs from standard two-stage CRS by dynamically bridging retrieval and generation. First, a retriever model generates candidate items based on user history; in the subsequent stage, an LLM refines the recommendations by incorporating conversational context with retrieved results. In addition, we introduce a novel reinforcement learning (RL) method that leverages LLM feedback to iteratively update the retriever. By creating a collaborative feedback loop that reinforces sampled candidate sets with higher ranking metrics, RAR effectively mitigates the misalignment between the retrieval and generation stages. Furthermore, grounding the LLM in factual metadata allows our RL-driven approach to capture subtle user intentions and generate context-aware recommendations with reduced hallucinations. We validate our approach through extensive experiments on multiple benchmarks, where RAR consistently outperforms state-of-the-art baseline methods.

Problem

• Conversational Recommendation
• Retrieval Augmentation
• Large Language Models
• Reinforcement Learning
• Factuality

Innovation

• Retrieval-Augmented Generation
• Conversational Recommendation
• Reinforcement Learning
• Large Language Models
• Factuality