Precise Legal Sentence Boundary Detection for Retrieval at Scale: NUPunkt and CharBoundary

2025-04-05

AI Summary

Legal texts pose challenges for general-purpose sentence boundary detection (SBD) due to domain-specific citations, abbreviations, and complex syntactic structures. To address this, we propose a lightweight, legal-domain-specific SBD paradigm comprising two complementary models: NUPunkt—a rule-augmented, punctuation-driven parser—and CharBoundary—a character-level lightweight machine learning model. Both operate exclusively on CPU, require no GPU, and support deployment via scikit-learn or ONNX. NUPunkt achieves 91.1% precision, processes 10 million characters per second, and consumes only 432 MB RAM; CharBoundary enables tunable precision–recall trade-offs, attaining an F1-score of 0.782 on large-scale evaluation. Relative to generic SBD tools, our approach improves accuracy by 29–32%, substantially mitigating context fragmentation in legal information retrieval. The toolkit is open-sourced under the MIT License, available on PyPI, and scales to process millions of documents within minutes.

Technology Category

• Natural Language Processing: Summarization
• Machine Learning: Large Multimodal Models (LMMs)
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics

Abstract

We present NUPunkt and CharBoundary, two sentence boundary detection libraries optimized for high-precision, high-throughput processing of legal text in large-scale applications such as due diligence, e-discovery, and legal research. These libraries address the critical challenges posed by legal documents containing specialized citations, abbreviations, and complex sentence structures that confound general-purpose sentence boundary detectors. Our experimental evaluation on five diverse legal datasets comprising over 25,000 documents and 197,000 annotated sentence boundaries demonstrates that NUPunkt achieves 91.1% precision while processing 10 million characters per second with modest memory requirements (432 MB). CharBoundary models offer balanced and adjustable precision-recall tradeoffs, with the large model achieving the highest F1 score (0.782) among all tested methods. Notably, NUPunkt provides a 29-32% precision improvement over general-purpose tools while maintaining exceptional throughput, processing multi-million document collections in minutes rather than hours. Both libraries run efficiently on standard CPU hardware without requiring specialized accelerators. NUPunkt is implemented in pure Python with zero external dependencies, while CharBoundary relies only on scikit-learn and optional ONNX runtime integration for optimized performance. Both libraries are available under the MIT license, can be installed via PyPI, and can be interactively tested at https://sentences.aleainstitute.ai/. These libraries address critical precision issues in retrieval-augmented generation systems by preserving coherent legal concepts across sentences, where each percentage improvement in precision yields exponentially greater reductions in context fragmentation, creating cascading benefits throughout retrieval pipelines and significantly enhancing downstream reasoning quality.

Problem

• Detects legal sentence boundaries with high precision
• Handles specialized citations and complex legal structures
• Improves retrieval-augmented generation systems' context coherence

Innovation

• High-precision legal sentence boundary detection
• Optimized for large-scale legal text processing
• Efficient CPU-based performance without accelerators