ChatGPT Unveils Its Limits: Principles of Law Deliver Checkmate

2025-10-22

AI Summary

This study identifies systematic limitations of ChatGPT in deep legal text analysis: although it possesses domain-relevant knowledge, it fails to integrate reading comprehension, multi-step reasoning, and structured generation—particularly in precisely extracting Principles of Law (PoLs) from judicial opinions. To rigorously assess its capabilities, we design controlled experiments benchmarking ChatGPT against a rule-driven, regular-expression-based system—the first such comparison for PoL identification and extraction. Results demonstrate that ChatGPT’s cross-paragraph reasoning and holistic inference capabilities are significantly inferior to both human legal experts and the interpretable rule-based system. The work introduces “systematic reasoning capacity” as a critical bottleneck in deploying large language models for legal applications, empirically underscoring the irreplaceable role of human legal intelligence in principle abstraction, contextual modeling, and logical closure.

Technology Category

• Natural Language Processing: Sentiment Analysis, Stylistic Analysis, and Argument Mining
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

This study examines the performance of ChatGPT with an experiment in the legal domain. We compare the outcome with it a baseline using regular expressions (Regex), rather than focusing solely on the assessment against human performance. The study reveals that even if ChatGPT has access to the necessary knowledge and competencies, it is unable to assemble them, reason through, in a way that leads to an exhaustive result. This unveils a major limitation of ChatGPT. Intelligence encompasses the ability to break down complex issues and address them according to multiple required competencies, providing a unified and comprehensive solution. In the legal domain, one of the most crucial tasks is reading legal decisions and extracting key passages condensed from principles of law (PoLs), which are then incorporated into subsequent rulings by judges or defense documents by lawyers. In performing this task, artificial intelligence lacks an all-encompassing understanding and reasoning, which makes it inherently limited. Genuine intelligence, remains a uniquely human trait, at least in this particular field.

Problem

• ChatGPT struggles with legal reasoning and comprehensive analysis
• AI lacks ability to assemble knowledge for exhaustive legal solutions
• Extracting legal principles requires human-like understanding and reasoning

Innovation

• Used regex baseline for legal text analysis
• Identified ChatGPT's reasoning limitations in law
• Highlighted lack of comprehensive legal understanding