An Empirical Study of Coordination Mode as the First-Class Citizen in From-Scratch Multi-Agent Coding

2026-07-30

AI Summary

Existing multi-agent coding evaluation benchmarks overlook real-world development costs in time and economics, conflate reasoning with communication, and rely solely on superficial task completion. This work proposes MSEval—the first evaluation framework that treats collaboration topology as a core variable. Implemented via the LegoGent engine across ten real full-stack projects, MSEval orchestrates periodic synchronization and CI/CD deployment across ten distinct collaboration structures. A hierarchical TAgent auto-grader assesses performance jointly on functional correctness, latency, and reasoning cost using deterministic rules aligned with layered requirements. Experiments reveal that collaboration topology exerts an influence on the trade-off among speed, cost, and quality comparable to that of model capability itself, with score differences exceeding 30 points across structures; notably, the structured pipeline achieves the fastest convergence and highest quality.

Abstract

Multi-agent vibe coding promises to accelerate software development, yet existing benchmarks rely on synthetic environments that ignore practical time and monetary costs, conflate reasoning with communication, and reward only superficial completion. We introduce multi-agent from-scratch evaluation benchmark, MSEval, evaluating multi-agent coding on real-world tasks. Grounded in 10 authentic, full-stack projects across 10 domains, MSEval scores performance using hierarchical requirements and deterministic rubrics. Its execution engine, LegoGent, tests 10 collaboration topologies where agents coordinate via periodic sync intervals and deploy through native CI/CD pipelines. Concurrently, the automated grader TAgent dynamically probes implementations to jointly measure functional success, latency, and prefix-cached token cost. Across 100 runs, MSEval reveals that organizational topology rivals model capability in shaping the speed--cost--quality trade-off. For identical tasks and models, varying the topology shifts scores by over 30 points and doubles wall-clock time. Structured pipelines converge fastest with the highest quality, whereas heavy managerial oversight degrades performance. Ultimately, MSEval establishes a rigorous, reproducible standard for measuring how multi-agent teams actually build software. The benchmark is released at https://github.com/robinren03/MSEval.

Problem

• multi-agent coding
• evaluation benchmark
• coordination mode
• software development
• real-world tasks

Innovation

• multi-agent coding
• coordination topology
• from-scratch evaluation
• CI/CD integration
• empirical benchmarking