Knowledgeable-r1: Policy Optimization for Knowledge Exploration in Retrieval-Augmented Generation

2025-06-05

AI Summary

Current RAG systems over-rely on retrieved context, rendering them vulnerable to misleading or redundant information while neglecting the large language model’s intrinsic parametric knowledge. To address this, we propose a multi-strategy joint sampling framework that dynamically integrates parametric and contextual knowledge and resolves conflicts between them through strategy optimization, collaborative sampling across multiple distributions, and explicit knowledge capability modeling. Our approach introduces, for the first time in knowledge capability exploration, a learnable joint distribution mechanism. This significantly enhances RAG’s robustness against parametric–contextual conflicts: it achieves a 17.07% absolute accuracy gain on counterfactual reasoning tasks and delivers consistent performance improvements across diverse RAG benchmarks. The framework establishes a novel paradigm for balancing retrieval augmentation with model-internal knowledge, advancing the principled integration of parametric and external knowledge sources.

Technology Category

• Knowledge Representation and Reasoning: Knowledge Acquisition
• Search and Optimization: Metareasoning and Metaheuristics
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Retrieval-augmented generation (RAG) is a mainstream method for improving performance on knowledge-intensive tasks. However,current RAG systems often place too much emphasis on retrieved contexts. This can lead to reliance on inaccurate sources and overlook the model's inherent knowledge, especially when dealing with misleading or excessive information. To resolve this imbalance, we propose Knowledgeable-r1 that using joint sampling and define multi policy distributions in knowledge capability exploration to stimulate large language models'self-integrated utilization of parametric and contextual knowledge. Experiments show that Knowledgeable-r1 significantly enhances robustness and reasoning accuracy in both parameters and contextual conflict tasks and general RAG tasks, especially outperforming baselines by 17.07% in counterfactual scenarios and demonstrating consistent gains across RAG tasks. Our code are available at https://github.com/lcy80366872/ knowledgeable-r1.

Problem

• Balancing reliance on retrieved contexts and model's inherent knowledge
• Addressing inaccuracies from misleading or excessive information in RAG
• Enhancing robustness and reasoning accuracy in knowledge-intensive tasks

Innovation

• Joint sampling for knowledge exploration
• Multi-policy distributions in RAG
• Enhancing parametric and contextual knowledge integration