BriefMe: A Legal NLP Benchmark for Assisting with Legal Briefs

2025-06-07

AI Summary

This study addresses the practical utility of large language models (LLMs) in legal NLP, specifically for litigation brief drafting. To bridge the gap in rigorous evaluation, we introduce BRIEFME—the first multi-task benchmark tailored to legal brief writing—comprising argument summarization, argument completion, and case law retrieval. BRIEFME is the first to systematically model creative legal argumentation and precedent-based reasoning, supported by a practice-oriented, multidimensional evaluation framework. The dataset is constructed from human-annotated judicial decisions and authentic litigation briefs. Zero-shot and few-shot experiments reveal that LLMs outperform human-written headings in argument summarization and guided completion, yet exhibit significant deficiencies in generating substantive legal arguments and retrieving contextually relevant precedents—highlighting critical limitations in their capacity for deep legal reasoning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

A core part of legal work that has been under-explored in Legal NLP is the writing and editing of legal briefs. This requires not only a thorough understanding of the law of a jurisdiction, from judgments to statutes, but also the ability to make new arguments to try to expand the law in a new direction and make novel and creative arguments that are persuasive to judges. To capture and evaluate these legal skills in language models, we introduce BRIEFME, a new dataset focused on legal briefs. It contains three tasks for language models to assist legal professionals in writing briefs: argument summarization, argument completion, and case retrieval. In this work, we describe the creation of these tasks, analyze them, and show how current models perform. We see that today's large language models (LLMs) are already quite good at the summarization and guided completion tasks, even beating human-generated headings. Yet, they perform poorly on other tasks in our benchmark: realistic argument completion and retrieving relevant legal cases. We hope this dataset encourages more development in Legal NLP in ways that will specifically aid people in performing legal work.

Problem

• Assisting legal professionals in writing and editing briefs
• Evaluating language models on legal argument tasks
• Improving case retrieval and argument completion in Legal NLP

Innovation

• Introduces BRIEFME dataset for legal briefs
• Evaluates models on summarization and completion
• Highlights poor performance on case retrieval