🤖 AI Summary
Agent-based modeling (ABM) lacks systematic, standardized benchmarks for policy evaluation, hindering rigorous assessment of its capabilities in real-world policy analysis. Method: We introduce the first ABM capability benchmark specifically designed for policy evaluation, comprising 20 end-to-end policy modeling scenarios, 65 fine-grained subtasks, and 200 automatically generated tasks. Our hierarchical evaluation framework—structured as Scenario → Subtask → Generated Task—integrates multi-agent modeling, behavioral calibration, expert validation, and automated task generation, grounded in social simulation theory and empirical policy analysis frameworks. Contribution/Results: Experiments reveal that state-of-the-art ABM methods achieve only 24.5%, 15.04%, and 14.5% coverage across the three task categories, exposing a substantial gap between current ABM capabilities and practical policy assessment requirements. This benchmark establishes a reproducible, extensible evaluation infrastructure to guide future research and development in policy-oriented ABM.
📝 Abstract
With the growing adoption of agent-based models in policy evaluation, a pressing question arises: Can such systems effectively simulate and analyze complex social scenarios to inform policy decisions? Addressing this challenge could significantly enhance the policy-making process, offering researchers and practitioners a systematic way to validate, explore, and refine policy outcomes. To advance this goal, we introduce PolicySimEval, the first benchmark designed to evaluate the capability of agent-based simulations in policy assessment tasks. PolicySimEval aims to reflect the real-world complexities faced by social scientists and policymakers. The benchmark is composed of three categories of evaluation tasks: (1) 20 comprehensive scenarios that replicate end-to-end policy modeling challenges, complete with annotated expert solutions; (2) 65 targeted sub-tasks that address specific aspects of agent-based simulation (e.g., agent behavior calibration); and (3) 200 auto-generated tasks to enable large-scale evaluation and method development. Experiments show that current state-of-the-art frameworks struggle to tackle these tasks effectively, with the highest-performing system achieving only 24.5% coverage rate on comprehensive scenarios, 15.04% on sub-tasks, and 14.5% on auto-generated tasks. These results highlight the difficulty of the task and the gap between current capabilities and the requirements for real-world policy evaluation.