Are manual annotations necessary for statutory interpretations retrieval?

2025-06-16

AI Summary

This work investigates the necessity and optimal practices of human annotation in legal concept explanation retrieval. Addressing the bottleneck of existing methods—reliance on costly, concept-level manual annotations—we conduct three systematic empirical validations: (1) quantifying the minimal annotation requirement per legal concept; (2) comparing random sampling versus high-quality candidate sentence prioritization for annotation; and (3) evaluating the fidelity and efficiency of LLM-based (GPT-4/Claude) automatic annotation. Experiments employ BERT/RoBERTa semantic retrieval models within an active learning framework. Results show that only 5–10 carefully selected annotations per concept achieve 98% of the performance attained with full human annotation; LLM-generated annotations attain 94% of human annotators’ F1 score, substantially reducing annotation cost. This study provides the first empirical evidence that legal concept explanation retrieval can operate with dramatically reduced human annotation dependency, establishing a novel low-resource paradigm for legal AI.

Technology Category

• Machine Learning: Active Learning
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Natural Language Processing: Information Extraction

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

One of the elements of legal research is looking for cases where judges have extended the meaning of a legal concept by providing interpretations of what a concept means or does not mean. This allow legal professionals to use such interpretations as precedents as well as laymen to better understand the legal concept. The state-of-the-art approach for retrieving the most relevant interpretations for these concepts currently depends on the ranking of sentences and the training of language models over annotated examples. That manual annotation process can be quite expensive and need to be repeated for each such concept, which prompted recent research in trying to automate this process. In this paper, we highlight the results of various experiments conducted to determine the volume, scope and even the need for manual annotation. First of all, we check what is the optimal number of annotations per a legal concept. Second, we check if we can draw the sentences for annotation randomly or there is a gain in the performance of the model, when only the best candidates are annotated. As the last question we check what is the outcome of automating the annotation process with the help of an LLM.

Problem

• Assessing necessity of manual annotations for legal interpretations retrieval
• Determining optimal annotation volume per legal concept
• Evaluating automated annotation using LLMs versus manual methods

Innovation

• Optimizing manual annotation volume per legal concept
• Selecting best candidate sentences for annotation
• Automating annotation process using LLM technology