Auto Research for Materials: Auditable AI-Scientist Workflows with Held-Out Transfer

2026-07-19

AI Summary

Current AI research agents in materials modeling often suffer from poor generalization, auditability, and reusability due to over-optimization of internal metrics. This work proposes an auditable scientific workflow evaluation framework that decouples automated research pipelines into four types of modifications: features, models, representations, and training data. By integrating modular search strategies, five-fold inner-loop feedback, and a frozen held-out test set for unbiased evaluation, the study systematically assesses 701 distinct modifications across ten Matbench tasks. Results demonstrate that the top-performing modifications identified on nine tasks consistently outperform baselines on unseen data, achieving up to a 26.3% reduction in mean absolute error (MAE). Furthermore, the analysis reveals distinct effective improvement pathways for composition- versus structure-based tasks, enabling cross-task compositional reuse of successful components.

Abstract

An AI research agent can improve the score it sees without finding a modelling change that works on new materials. We ask a stricter question. After repeated experiments, does the selected change survive on data that never entered the loop, and can its code be reused? We separate the search into changes to features, models, representations, and training data. Seven searches produce 701 evaluated changes across ten Matbench endpoints. Agents receive only the mean over five inner folds, reducing reliance on any single development split. We then freeze the selected code and evaluate it once on an untouched holdout. Nine of ten choices remain the best tested single intervention. The surviving changes reveal two materials modelling regimes. With composition alone, feature, model, and representation changes provide comparable routes to improvement. They include held-out MAE reductions of 17.4\% for band gap and 18.6\% for steel strength, as well as gains on both classification endpoints, while screened external data adds little. For structure tasks, richer geometry descriptors and model or calibration changes lower mean held-out MAE by 14.6\% and 7.1\% and lead on different property families, whereas composition embeddings do not transfer. Combining separately found feature and model changes yields a 26.3\% mean held-out improvement. These results show in materials prediction that closed-loop agents can produce decisions that survive unseen evidence and code changes that can be reused across tasks and combined. More broadly, they provide an evaluation design for testing executable discoveries beyond the feedback loop.

Problem

• materials informatics
• AI research agent
• holdout evaluation
• transferability
• auditable AI

Innovation

• auditable AI-scientist
• held-out transfer
• closed-loop optimization
• materials informatics
• reusable code changes