Asuka-Bench: Benchmarking Code Agents on Underspecified User Intent and Multi-Round Refinement

2026-06-04

AI Summary

Existing code generation benchmarks evaluate only single-turn mappings from static prompts to outputs, failing to capture the ambiguous user intent and iterative refinement characteristic of real-world web development. To address this limitation, this work proposes the first browser-rendering-based, multi-turn interactive benchmark, establishing a closed-loop optimization pipeline where a Code Agent generates code, a UI Agent performs automated testing, and a User LLM simulates human feedback. This approach uniquely integrates fuzzy intent modeling with multi-round refinement and introduces automatic evaluation grounded in actual UI behavior. Evaluated on 50 web tasks across eight prominent LLMs, the benchmark reveals a weighted pass-rate gap of up to 38 percentage points; even the strongest model completes only 52% of tasks after three iterations, demonstrating the benchmark’s challenge and indicating significant room for improvement.

Abstract

Existing code-generation benchmarks score a single mapping from a complete prompt to a one-shot output. However, real web development is different. Users seldom write a full spec at the start; many requirements only become clear once they look at an intermediate result and react to it. We present Asuka-Bench, a benchmark that pairs underspecified user intent with multi-round refinement, grounded in browser-rendered behavior. Each task is resolved through a closed loop: a Code Agent generates a web project, a UI Agent executes test cases on the deployed site, and a User LLM turns evaluation outcomes into natural-language feedback for the next round. The benchmark comprises 50 web tasks with 784 evaluation criteria and 2402 expected outcomes. We benchmark 8 LLMs across 2 agent frameworks. The results separate models clearly: weighted Task Pass Rate varies by 38 percentage points and models also differ substantially in their ability to repair from feedback. Asuka-Bench is also far from saturated: even the strongest model completes only 52% of projects after three rounds.

Problem

• code generation
• underspecified intent
• multi-round refinement
• user feedback
• web development

Innovation

• code agents
• underspecified intent
• multi-round refinement
• browser-rendered evaluation
• interactive feedback loop