LegalPincite: Multi-level Legal Information Retrieval Dataset

2026-08-04

AI Summary

This work addresses critical limitations in existing legal information retrieval benchmarks—namely query data leakage, incomplete corpus coverage, and the absence of authentic paragraph-level citation annotations—which collectively undermine evaluation validity. To remedy these issues, the authors introduce LegalPincite, the first large-scale legal retrieval benchmark derived from Court of Justice of the European Union case law. Through rigorous case preprocessing, citation relation extraction, query masking, and expert validation, LegalPincite provides a leakage-free, comprehensively covered corpus with multi-granularity citation annotations at the paragraph level. The dataset enables multi-tiered retrieval tasks spanning case-to-case, paragraph-to-case, and paragraph-to-paragraph levels, substantially enhancing both the realism and challenge of evaluating legal retrieval models.

Abstract

A common task in legal Information Retrieval (IR) is to find relevant legal sources from case-law collections. While legal practice often requires pinpoint citations (pincites) to specific case paragraphs, most existing public legal IR datasets lack paragraph-level citation annotations. Yet, publicly available datasets with such information contain data leakage in the query text and exclude paragraphs that are neither citing nor cited from the corpora, creating an unrealistic and oversimplified retrieval setting, potentially leading to inflated performance. To address these limitations, we contribute a large-scale legal IR dataset constructed from Court of Justice of the European Union (CJEU) judgments. The dataset contains: (i) masked case/paragraph queries, with removed citation information; (ii) a corpus that includes all paragraphs; and (iii) case- and paragraph-level ground-truth citations, with partial human expert validation. Our dataset supports both the development and rigorous evaluation of legal IR methods, at multiple query-document levels (case-to-case, paragraph-to-case, and paragraph-to-paragraph retrieval). Link to dataset: https://huggingface.co/datasets/theresiavr/legalpincite

Problem

• legal information retrieval
• pinpoint citations
• paragraph-level retrieval
• data leakage
• legal IR dataset

Innovation

• legal information retrieval
• pinpoint citations
• multi-level retrieval
• dataset construction
• citation annotation