FEM-Bench: A Structured Scientific Reasoning Benchmark for Evaluating Code-Generating LLMs
career value
218K/year
🤖 AI Summary
Current LLM evaluations lack scientifically valid benchmarks tailored to computational mechanics, hindering rigorous assessment of their ability to generate physically consistent and numerically reliable finite element method (FEM) code. To address this, we propose FEM-Bench—the first structured scientific reasoning benchmark that systematically integrates physical principles (e.g., partial differential equations, constitutive relations) with numerical constraints (e.g., mesh discretization, element assembly), covering core modeling tasks including geometric representation, material behavior, and boundary condition specification. We introduce a novel “physics–mathematics” dual-constraint criterion for verifiable scientific code generation, enabling progressive difficulty scaling and objective quantification. Coupled with an automated unit-testing framework, FEM-Bench evaluates 33 nontrivial introductory tasks: Gemini 3 Pro achieves a 78.8% full-pass rate in function generation, while GPT-5 attains a 73.8% pass rate in joint testing—revealing critical limitations of current LLMs in rigorous scientific programming.
Application Category
📝 Abstract
As LLMs advance their reasoning capabilities about the physical world, the absence of rigorous benchmarks for evaluating their ability to generate scientifically valid physical models has become a critical gap. Computational mechanics, which develops and applies mathematical models and numerical methods to predict the behavior of physical systems under forces, deformation, and constraints, provides an ideal foundation for structured scientific reasoning evaluation. Problems follow clear mathematical structure, enforce strict physical and numerical constraints, and support objective verification. The discipline requires constructing explicit models of physical systems and reasoning about geometry, spatial relationships, and material behavior, connecting directly to emerging AI goals in physical reasoning and world modeling. We introduce FEM-Bench, a computational mechanics benchmark designed to evaluate the ability of LLMs to generate correct finite element method (FEM) and related code. FEM-Bench 2025 contains a suite of introductory but nontrivial tasks aligned with material from a first graduate course on computational mechanics. These tasks capture essential numerical and physical modeling challenges while representing only a small fraction of the complexity present in the discipline. Despite their simplicity, state-of-the-art LLMs do not reliably solve all of them. In a five attempt run, the best performing model at function writing, Gemini 3 Pro, completed 30/33 tasks at least once and 26/33 tasks all five times. The best performing model at unit test writing, GPT-5, had an Average Joint Success Rate of 73.8%. Other popular models showed broad performance variation. FEM-Bench establishes a structured foundation for evaluating AI-generated scientific code, and future iterations will incorporate increasingly sophisticated tasks to track progress as models evolve.
Problem
Research questions and friction points this paper is trying to address.
Evaluates LLMs' ability to generate scientifically valid physical models.
Assesses code generation for finite element method and related computational mechanics tasks.
Provides a structured benchmark for AI-generated scientific code in physical reasoning.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Introduces FEM-Bench for evaluating LLMs in computational mechanics
Uses finite element method code generation tasks for assessment
Provides structured benchmark with objective verification of scientific reasoning
🔎 Similar Papers
No similar papers found.
