Can Large Language Models Understand As Well As Apply Patent Regulations to Pass a Hands-On Patent Attorney Test?

2025-07-11

AI Summary

This study systematically evaluates the legal expertise of large language models (LLMs) in the context of the European Qualifying Examination (EQE) for patent attorneys, focusing on their depth of understanding and practical application of patent law. Method: We benchmark prominent open- and closed-weight models—including GPT-series, Claude, DeepSeek, and Llama-3—using a comprehensive evaluation framework integrating text generation, multimodal input processing, and expert human adjudication across three dimensions: legal reasoning, cross-document integration, and output stability. Contribution/Results: We uncover a significant misalignment between automated scoring and expert judgment, establishing legal reasoning quality—not mere answer correctness—as the critical evaluation metric. We further identify high sensitivity to temperature settings and prompt engineering. Experimental results show OpenAI o1 achieves the highest accuracy (0.82), yet all models fall short of the passing threshold (0.90), empirically confirming that current LLMs cannot substitute for qualified human patent attorneys and establishing a rigorous benchmark for legal AI capabilities.

Technology Category

Machine Learning: Large Multimodal Models (LMMs)Natural Language Processing: (Large) Language ModelsComputer Vision: Large Vision Models

Application Category

Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved informationEconomics, Online Markets and Human Computation: Cost models of using LLMs in production systemsSemantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

The legal field already uses various large language models (LLMs) in actual applications, but their quantitative performance and reasons for it are underexplored. We evaluated several open-source and proprietary LLMs -- including GPT-series, Anthropic, Deepseek and Llama-3, variants -- on parts of the European Qualifying Examination (EQE) for future European Patent Attorneys. OpenAI o1 led with 0.82 accuracy and 0.81 F1 score, whereas (Amazon Web Services) AWS Llama 3.1 8B lagged at 0.50 accuracy, and a Python-deployed Llama 3.1 8B scored 0.55. The latter two are within the range of mere guessing for the two-answer forced-choice design. None of the evaluated models could have passed the examination fully, as accuracy never exceeded the average threshold of 0.90 required for professional-level standards -- also not models that are regularly promoted for their assumed beyond-PhD- and bar-admitted-lawyer-level performance. GPT-4o excelled at integrating text and graphics, while Claude 3 Opus often lost formatting coherence. Human patent experts evaluated the textual justifications and uncovered various critical shortcomings of each model. They valued clarity and legal rationale over the raw correctness of the answers, which revealed misalignment between automatic metrics and expert judgment. Model outputs were sensitive to modest temperature changes and prompt wording, which underscores the remaining necessity of expert oversight. Future work should target logical consistency, robust multimodality, and adaptive prompting to approach human-level patent proficiency. In summary, despite the outstanding performance of recent large models, the general public might overestimate their performance. The field has a long way to go to develop a virtual patent attorney. This paper wants to point out several specific limitations that need solutions.

Problem

• Evaluate LLMs' ability to understand and apply patent regulations
• Assess performance gaps in LLMs for patent attorney tests
• Identify shortcomings in model outputs and expert alignment

Innovation

• Evaluated LLMs on European Patent Attorney Exam
• GPT-4o excels in text-graphics integration
• Human experts prioritize clarity over raw correctness