CraftBench-UE: Deterministic Evaluation for Coding Agents in Unreal Engine

2026-09-19

AI Summary

本文介绍了CraftBench-UE，一种在Unreal Engine中对编码代理进行确定性评估的工具，通过重建提交、执行检查来确保游戏功能正确实现，并比较了C++和Blueprint任务完成率。

Abstract

Building gameplay features in a game engine requires more than code, as code that compiles and runs does not necessarily implement the requested gameplay. We introduce CraftBenchUE, an evaluation harness that runs agents in an isolated Unreal Engine environment, reconstructs their saved submissions in fresh projects, and applies deterministic build, asset, and runtime checks without an LLM judge. Based on the harness, we built a benchmark consisting of 70 tasks spanning C++ source, Blueprint assets, and editor scripting. We evaluate seven models under two editor-tool configurations, with a file-and-shell baseline on C++ tasks. We further pair tasks that specify the same gameplay and use the same runtime tests, but require C++ and Blueprint as the deliverables. Across the 10 paired tasks, C++ completion rates exceed Blueprint by 30.0 and 42.9 percentage points in the two tool configurations. Among on-time Blueprint submissions in this paired set that pass asset checks, 42.2% and 50.0% fail explicit runtime assertions. These submissions satisfy asset requirements but fail the required gameplay tests. We will release the harness, task benchmark, and our trajectory findings with the report.

Problem

• gameplay
• evaluation
• coding agents
• Unreal Engine
• deterministic

Innovation

• deterministic evaluation
• Unreal Engine
• gameplay features
• coding agents
• benchmark