AgentWorld: Benchmarking Long-Horizon Collaboration of Multi-agent LLMs

2026-09-25

AI Summary

This study addresses the challenge of effectively evaluating the collaborative capabilities of large language model agents in long-horizon, real-world scenarios. To this end, it proposes a multi-agent collaboration benchmark built upon an MMORPG sandbox environment, comprising 100 tasks that require three to twenty asymmetrically role-assigned agents to complete over fifty interaction rounds through communication and planning. Furthermore, this work introduces the Causal Collaboration Effectiveness (CCE) metric, which leverages graph-structured action dependency tracking to quantify individual contributions, thereby overcoming the limitations of traditional binary success-or-failure evaluations. Experimental results reveal that even the best-performing models achieve only a 52% success rate, exposing critical failure modes such as communication breakdowns. The associated code and datasets have been fully open-sourced.

Abstract

Existing multi-agent benchmarks primarily test in competitive settings, short-horizon interactions under 20 steps, or simply aggregate individual performance, failing to isolate and highlight genuine collaboration capabilities of LLM-based agents. We introduce AgentWorld, a benchmark of 100 human-annotated tasks (with 100 augmented variants) for evaluating long-horizon, multi-agent collaboration. Tasks span 50+ interaction rounds across a rich MMORPG sandbox and require 3-20 agents with asymmetric roles and abilities to coordinate through communication, joint planning, and resource sharing under a blackbox setting where each agent acts independently without access to others' internal states. To quantify collaboration effectiveness in addition to conventional binary task success, we propose Causal Collaboration Effectiveness (CCE), a graph-based metric that traces causal dependencies between agent actions and measures what fraction of a team's effort actually contributed to the outcome. Experiments with Gemini 3 Flash, Claude Haiku 4.5, GPT-5 Mini, and DeepSeek R1-70B show that even the best model achieves only 52.0% task success, with systematic failure modes including communication breakdowns, role confusion, and inability to maintain shared plans across rounds. AgentWorld is fully open-source.

Problem

• Multi-agent LLMs
• Long-horizon collaboration
• Benchmark
• Collaboration effectiveness
• MMORPG sandbox

Innovation

• Multi-agent collaboration
• Long-horizon benchmark
• Causal Collaboration Effectiveness
• MMORPG sandbox
• Blackbox setting