Agents on the Bench: Large Language Model Based Multi Agent Framework for Trustworthy Digital Justice

2024-12-24

AI Summary

Judicial AI systems frequently suffer from low transparency and poor interpretability, undermining public trust. To address this, we propose AgentsBench—a novel, first-of-its-kind LLM-based multi-agent deliberative framework tailored to judicial settings, which emulates the functional division of labor, legal reasoning, and consensus-building processes of a collegiate bench. Our approach innovatively integrates role-aware prompting, consensus-driven decision-making mechanisms, and task-specific modeling for legal judgment prediction. Evaluated on multi-charge judgment prediction tasks, AgentsBench significantly outperforms existing LLM-based methods in accuracy, fairness—particularly in mitigating cross-group disparities—and modeling of socio-legal factors, while demonstrating strong generalization across diverse case types. This work establishes a new paradigm for trustworthy, deliberative AI in judicial decision support.

Technology Category

• Philosophy and Ethics of AI: AI & Law, Justice, Regulation & Governance
• Multiagent Systems: Mechanism Design
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

The justice system has increasingly employed AI techniques to enhance efficiency, yet limitations remain in improving the quality of decision-making, particularly regarding transparency and explainability needed to uphold public trust in legal AI. To address these challenges, we propose a large language model based multi-agent framework named AgentsBench, which aims to simultaneously improve both efficiency and quality in judicial decision-making. Our approach leverages multiple LLM-driven agents that simulate the collaborative deliberation and decision making process of a judicial bench. We conducted experiments on legal judgment prediction task, and the results show that our framework outperforms existing LLM based methods in terms of performance and decision quality. By incorporating these elements, our framework reflects real-world judicial processes more closely, enhancing accuracy, fairness, and society consideration. AgentsBench provides a more nuanced and realistic methods of trustworthy AI decision-making, with strong potential for application across various case types and legal scenarios.

Problem

• Artificial Intelligence
• Transparency
• Explainability

Innovation

• AgentsBench
• AI Transparency
• Judicial Decision-making