🤖 AI Summary
This work proposes the first framework that formalizes the discovery of structural mappings from statistical mechanics solvable models as an AI agent task. Addressing the challenge of automatically determining whether a given partition function can be mapped to a known solvable Ising-type model, the authors introduce StatMechBench-v0—a benchmark comprising six problem classes—and design a multi-layer verification mechanism integrating large language models, numerical simulations, symbolic computation, and Pfaffian analysis. Experiments reveal that while current LLM-based agents can leverage feedback to correct code and recover partition functions, they frequently misclassify solvability categories or underestimate problem complexity. These findings highlight fundamental limitations in structural reasoning by existing models and underscore the necessity of incorporating symbolic reasoning and structural invariant verification into AI-driven scientific discovery pipelines.
📝 Abstract
An important skill in theoretical physics is to recognize when a new problem can be transformed into a known model. We study this skill as an AI-agent task: can LLM-based agents discover statistical mechanical mappings from a raw partition function to a tractable representation? To probe this question, we introduce StatMechBench-v0, a benchmark of six Ising-type problems covering transfer-matrix methods, gauge-removable disorder, and planar/Pfaffian structure. We evaluate a simple propose-verify-revise agent across multiple LLMs and problem phrasings. The results show that numerical feedback often helps agents repair code and recover correct partition functions. However, agents can also pass the numerical checks while misidentifying the underlying tractable class or understating computational complexity. This both reveals limitations in current LLM reasoning and calls for a verification stack that goes beyond numerical agreement, incorporating, for example, symbolic checks and structural invariants. Our study provides an early evaluation and design directions for AI agents aimed at structural discovery in theoretical physics.