EnterpriseRAG-Bench: A RAG Benchmark for Company Internal Knowledge

2026-05-05

AI Summary

This work addresses the inadequacy of existing RAG benchmarks in capturing the characteristics of enterprise-specific, multi-source, unstructured knowledge, which limits their ability to evaluate AI agents in complex real-world business settings. To bridge this gap, we introduce the first enterprise-grade RAG benchmark, comprising approximately 500,000 synthetic documents and 500 questions spanning nine enterprise data sources—including Slack, Gmail, and GitHub—and ten reasoning tasks. The benchmark supports scenarios ranging from single-document retrieval to multi-document conflict resolution, incorporating realistic noise such as cross-document inconsistency, misplaced documents, near-duplicates, and factual contradictions. We also provide a customizable synthetic data generation framework that enables variant creation by industry, organization size, and data source composition. The full dataset, evaluation toolkit, and leaderboard are publicly released to establish a standardized testbed for developing AI agents grounded in enterprise knowledge.

Abstract

Retrieval-Augmented Generation (RAG) has become the standard approach for grounding large language models in information that was not available during training. While existing datasets and benchmarks focus on web or other public sources, there is still no widely adopted dataset that realistically reflects the nature of company-internal knowledge. Meanwhile, startups, enterprises, and researchers are increasingly developing AI Agents designed to operate over exactly this kind of proprietary data. To close this gap, we release a synthetic enterprise corpus, its generation framework, and a leaderboard. We present EnterpriseRAG-Bench, a dataset consisting of approximately 500,000 documents spanning nine enterprise source types (Slack, Gmail, Linear, Google Drive, HubSpot, Fireflies, GitHub, Jira, and Confluence) and 500 questions across ten categories that test distinct retrieval and reasoning capabilities. The corpus is generated with cross-document coherence (grounded in shared projects, people, and initiatives) and augmented with realistic noise such as misfiled documents, near-duplicates, and conflicting information. The question set ranges from simple single-document lookups to multi-document reasoning, constrained retrieval, conflict resolution, and recognizing when information is absent. The generation framework lets teams generate variants tailored to their own industry, scale, and source mix. The dataset, code, evaluation harness, and leaderboard are available at https://github.com/onyx-dot-app/EnterpriseRAG-Bench.

Problem

• Retrieval-Augmented Generation
• enterprise knowledge
• RAG benchmark
• internal data
• proprietary information

Innovation

• EnterpriseRAG-Bench
• synthetic enterprise corpus
• cross-document coherence
• realistic noise injection
• multi-document reasoning