ExplorationBench: Measuring AI Systems' Exploration in Verifiable Alien Worlds

2026-09-24

AI Summary

This study addresses the challenge of disentangling genuine scientific exploration capabilities from pre-trained knowledge memorization in existing AI evaluations. To this end, it introduces a verifiable extraterrestrial world benchmark comprising dual sandboxes, AlienCode and AlienLogic. By establishing counterintuitive rule environments to eliminate memorization confounds, and employing erroneous manuals, environmental feedback, and tool-calling architectures, the framework compels AI systems to conduct authentic hypothesis testing. Evaluations across ten AI systems reveal that while leading models can acquire novel rules, their performance exhibits significant fluctuations, and sustained exploration frequently stagnates or degrades. These findings expose critical bottlenecks in current AI capabilities for autonomous scientific discovery.

Abstract

Scientific discovery begins where known problems end. There, AI systems must engage in exploration: framing hypotheses, designing experiments, and iterating on the results. However, evaluating this ability is difficult: (1) how to verify whether a genuinely new hypothesis holds, and (2) how to determine whether a system has discovered it through exploration or merely recalled related knowledge from pre-training data. To this end, we introduce ExplorationBench, which turns the wicked problem of evaluating scientific exploration into a concrete and tractable framework built on verifiable Alien Worlds: their rules are executable, so every answer can be checked exactly, and they conflict with familiar knowledge, so recall alone cannot solve the tasks. The benchmark contains two sandboxes, AlienCode (31 discovery targets, 70 tasks) and AlienLogic (24 discovery targets, 70 tasks). Each sandbox provides a flawed manual, task-specific environmental feedback, and a dedicated tool-call schema. Systems use these resources to explore the sandbox, then solve held-out tasks. We evaluate 10 AI systems and find that the strongest systems can acquire and apply unfamiliar rules, while performance varies substantially across trajectories and continued exploration can stall or reverse earlier gains. ExplorationBench represents a step towards AI systems that can acquire and apply genuinely new knowledge through exploration in unknown environments.

Problem

• scientific exploration
• AI evaluation
• knowledge discovery
• verifiable environments
• benchmark

Innovation

• ExplorationBench
• Verifiable Alien Worlds
• Scientific Exploration
• Knowledge Acquisition
• AI Evaluation