CanLegalRAGBench: Evaluating Retrieval-Augmented Generation on Canadian Case Law

2026-05-28

AI Summary

This study addresses the susceptibility of existing legal Retrieval-Augmented Generation (RAG) systems to hallucination in the context of Canadian case law and the absence of evaluation benchmarks grounded in authentic legal queries. The authors introduce the first realistic question-answering benchmark tailored to the Canadian legal system, comprising expert-annotated answers along with supporting case law citations. They conduct a comparative evaluation of both open-source and closed-source embedding models, employing fine-grained analysis of answer supportiveness. Their findings reveal that open-source embeddings can match the performance of closed-source counterparts; however, 8–29% of generated claims lack grounding in retrieved documents, exposing a critical flaw in current automatic evaluation methods that erroneously treat alternative relevant documents as valid support.

Abstract

RAG-based legal assistants have been growing in popularity, but LLM hallucinations remain a key issue and potentially undermines justice. While benchmarks have been developed to evaluate progress, many rely on synthetic queries rather than realistic legal scenarios. Moreover, Canadian law remains underrepresented in existing evaluations. To address this gap, we introduce CanLegalRAGBench, a Canadian legal QA benchmark based on realistic queries and expert-annotated answers grounded in case law. Our evaluation shows that retrieval performance is sensitive to design choices and that open-source embedding models are competitive with closed source models. However, it also reveals the limitation of automatic evaluations that penalize systems for retrieving alternative relevant documents. We also find that generated answers often diverge from gold responses, either with hallucinations or by producing overly detailed or irrelevant content, with 8-29% of claims not being supported by the retrieved documents. We hope this benchmark will help drive continued progress in addressing limitations of legal RAG systems.

Problem

• legal RAG
• hallucination
• Canadian case law
• retrieval-augmented generation
• evaluation benchmark

Innovation

• legal RAG
• Canadian case law
• realistic legal queries
• expert-annotated benchmark
• hallucination evaluation