LawFlow : Collecting and Simulating Lawyers' Thought Processes

2025-04-26

AI Summary

Existing legal AI systems predominantly target isolated subtasks, failing to support end-to-end, high-stakes decision-making in real-world scenarios. To address this gap, we introduce LawFlow—the first dynamic, modular, and iterative legal workflow dataset—grounded in law students’ authentic reasoning during corporate formation tasks. LawFlow uniquely captures three hallmarks of legal practice: ambiguity, iterative revision, and client-adaptive reasoning. Through comparative human-AI process tracing, we identify systematic deficiencies in large language models regarding closed-loop decision-making and execution flexibility. We propose a collaborative AI paradigm centered on hybrid planning, adaptive execution, and decision-point support. Empirical findings show practitioners prefer AI in auxiliary roles—e.g., ideation, blind-spot identification, and solution generation—over autonomous decision-making. All data and code are open-sourced to advance explainable, human-in-the-loop legal AI. (149 words)

Technology Category

Humans and AI: Learning Human Values and PreferencesPhilosophy and Ethics of AI: AI & Law, Justice, Regulation & GovernanceNatural Language Processing: (Large) Language Models

Application Category

Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd workSemantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactionsSearch and Retrieval-Augmented AI: Agentic search

Abstract

Legal practitioners, particularly those early in their careers, face complex, high-stakes tasks that require adaptive, context-sensitive reasoning. While AI holds promise in supporting legal work, current datasets and models are narrowly focused on isolated subtasks and fail to capture the end-to-end decision-making required in real-world practice. To address this gap, we introduce LawFlow, a dataset of complete end-to-end legal workflows collected from trained law students, grounded in real-world business entity formation scenarios. Unlike prior datasets focused on input-output pairs or linear chains of thought, LawFlow captures dynamic, modular, and iterative reasoning processes that reflect the ambiguity, revision, and client-adaptive strategies of legal practice. Using LawFlow, we compare human and LLM-generated workflows, revealing systematic differences in structure, reasoning flexibility, and plan execution. Human workflows tend to be modular and adaptive, while LLM workflows are more sequential, exhaustive, and less sensitive to downstream implications. Our findings also suggest that legal professionals prefer AI to carry out supportive roles, such as brainstorming, identifying blind spots, and surfacing alternatives, rather than executing complex workflows end-to-end. Building on these findings, we propose a set of design suggestions, rooted in empirical observations, that align AI assistance with human goals of clarity, completeness, creativity, and efficiency, through hybrid planning, adaptive execution, and decision-point support. Our results highlight both the current limitations of LLMs in supporting complex legal workflows and opportunities for developing more collaborative, reasoning-aware legal AI systems. All data and code are available on our project page (https://minnesotanlp.github.io/LawFlow-website/).

Problem

• Capturing end-to-end legal workflows for real-world practice
• Comparing human and LLM reasoning in legal decision-making
• Designing AI assistance for modular, adaptive legal tasks

Innovation

• LawFlow captures dynamic legal reasoning processes
• Compares human and LLM-generated legal workflows
• Proposes hybrid AI-human legal workflow design