From Scientific Observations to Mechanisms: Benchmarking Hypothesis Generation by AI Scientists

2026-10-04

AI Summary

This study addresses the lack of effective evaluation frameworks for assessing the ability of AI scientists to translate empirical data into mechanistic hypotheses. To this end, we introduce MechHypoBench, the first benchmark for mechanistic hypothesis generation, which integrates multidisciplinary literature with large-scale real-world data. Furthermore, we propose an open-ended hypothesis evaluation framework grounded in consequences under hidden conditions. By leveraging cross-disciplinary knowledge integration, open hypothesis generation, and causal inference evaluation techniques, this work systematically examines the capacity of AI systems to distill underlying mechanisms from observations. Experimental results demonstrate that a significant gap persists between the hypotheses generated by current general-purpose agents or AI scientists and the ground-truth mechanisms, thereby highlighting both the challenges and the potential of this research direction.

Abstract

Data-driven mechanistic hypotheses are essential to scientific discovery because they explain how underlying processes produce observed phenomena. AI agents and AI scientists increasingly support scientific data analysis. However, their ability to turn empirical findings into mechanistic hypotheses remains insufficiently examined. To address this gap, we introduce MechHypoBench, the first benchmark for evaluating whether AI agents and AI scientists can generate such hypotheses from empirical data. It combines paper-derived mechanisms from 14 scientific fields with real-world datasets containing 17.98 million records. The construction retains the observational complexity of empirical data while providing a specified underlying mechanism. Agents analyze the observations and propose open-form hypotheses. We develop an evaluation framework that assesses open-form mechanistic hypotheses through their consequences under withheld conditions. Experiments with general agents and AI scientists reveal a substantial gap between generated hypotheses and the underlying mechanisms.

Problem

• Mechanistic Hypothesis Generation
• AI Scientists
• Benchmarking
• Scientific Discovery
• Empirical Data

Innovation

• MechHypoBench
• Mechanistic Hypotheses
• AI Scientists
• Evaluation Framework
• Benchmarking