MirrorCraft: Paired Evaluation under Hidden Rule Changes in Minecraft

2026-07-31

AI Summary

Existing Minecraft agent evaluation benchmarks are constrained by fixed game mechanics, limiting their ability to assess agent adaptability under implicit rule changes. This work proposes MirrorCraft, a paired mirror-world evaluation framework that introduces hidden rule modifications via server-side interventions (implemented through datapacks) while preserving terrain and task consistency. The study further introduces the Rule Intervention Effect metric to quantify the impact of such rule changes on agent performance. Leveraging the Mineflayer interface, the authors evaluate six types of agents—including ReAct—using deterministic milestones and success rates. Experimental results demonstrate that different rule sets exert significant effects on performance: ReAct achieves the best results in the absence of explicit rule descriptions, while providing precise rule specifications yields only marginal improvements.

Abstract

With the prosperity of the large language models (LLMs), it has become an interesting topic: how do LLM-based agents work in Minecraft? Unfortunately, most existing benchmarks evaluate them under fixed game mechanics. High performance in these settings does not show whether an agent can continue making progress when familiar recipes, drops, and other rules change. In this paper, we introduce MirrorCraft, a paired benchmark for evaluating agents under hidden rule changes in Minecraft. Each Mirror world is a copy of its paired Vanilla world, with selected server-side rules modified by the corresponding datapack. Terrain, spawn, resource placement, objective, interface, and action budget remain matched within every Vanilla-Mirror pair. MirrorCraft includes five controlled biomes, six rule suites, three progression objectives, two model families, and six agent configurations under a shared Mineflayer interface. We evaluate task progress with deterministic advancement milestones and success rate and use the Rule Intervention Effect (RIE) to measure the performance change between matched Vanilla and Mirror worlds. The experiments show that hidden rule changes have strongly different effects across suites. Among the configurations evaluated without rule descriptions, ReAct achieves the highest pooled Mirror score. Providing the exact rules yields modest gains in average progress and completion across all three objectives. MirrorCraft extends Minecraft evaluation beyond fixed mechanics and provides a controlled setting for studying how agents use gameplay outcomes when the rules of the current world differ from familiar ones.

Problem

• hidden rule changes
• Minecraft
• agent evaluation
• game mechanics
• LLM-based agents

Innovation

• MirrorCraft
• hidden rule changes
• paired benchmark
• Rule Intervention Effect
• LLM-based agents