foundation model evaluation

This study addresses the hardware platform dependency that hinders fair evaluation of foundation models’ reasoning capabilities, introducing the first infrastructure-agnostic, cross-platform benchmark. We systematically evaluate 15 state-of-the-art models on 79 reasoning problems spanning eight academic domains—including physics and mathematics—across three heterogeneous environments: a high-performance computing system (MareNostrum 5), a cloud platform (Nebius AI Studio), and an academic cluster (8×H200). Employing a multi-stage experimental design—baseline establishment, platform validation, and extended assessment—we find that training data quality exerts a significantly stronger influence on reasoning performance than parameter count, challenging the “bigger is better” assumption. The benchmark enables reproducible, cross-platform evaluation and longitudinal model tracking, providing empirically grounded guidance for model selection in education, research, and industry applications.

Amid the growing representational capacity of foundation models, this work examines the necessity and evolving role of explicit symbolic reasoning. Grounded in the principle of compression, it proposes a modeling–reasoning trade-off theory: symbolic reasoning functions not as an intrinsic component of intelligence but as a compensatory mechanism for information loss in simplified models. As models increasingly approximate reality, reliance on such symbolic scaffolding naturally diminishes. Through theoretical analysis, computational modeling, and an AI-philosophical framework, the study offers a unified account of the historical significance of symbolic methods and the empirical success of modern large language models. It further repositions symbolic reasoning’s future value—not as a core reasoning engine, but as an interpretable interface in human–AI interaction, crucial for enabling human oversight, verification, and trust calibration.

In model-based systems engineering, low experimental data reuse efficiency and excessive redundant experiments hinder digital engineering agility. To address this, this paper proposes a case-based reasoning (CBR)-driven experimental management framework that explicitly integrates domain knowledge. The framework features structured experimental metadata modeling, digital twin–enabled scenario semantic alignment, and an interpretable similarity assessment mechanism to intelligently determine whether historical experiments can be transferred to address new verification queries. Its key innovation lies in embedding domain knowledge explicitly into both the CBR retrieval and adaptation stages, thereby enabling trustworthy cross-operating-condition and cross-configuration experimental data reuse. Evaluated on an industrial-scale vehicle energy system design case, the framework reduces redundant experiments by 37% and shortens early verification cycles by 42% on average, significantly enhancing iterative efficiency in digital engineering and advancing intelligent experimental management.

This study addresses the challenge of effectively guiding students to transition from misconceptions to accurate causal reasoning within a single classroom session, particularly in engineering fault-diagnosis contexts. It proposes a “detective-style scaffolding instructional framework” that reconfigures classroom voting systems into evidence-centered reasoning probes—rather than mere engagement tools—through three stages: hypothesis activation, evidence structuring, and causal integration. The approach exposes how conventional scoring often misjudges reasoning quality and instead employs dual-precision reasoning analysis to assess learning outcomes. In experiments, 80 third-year polymer engineering students increased their correct identification rate of humidity as the root cause from 29% to 100% within 90 minutes; additionally, 26 high school students without engineering backgrounds achieved 100% accuracy on transfer tasks and demonstrated significantly enhanced confidence in data analysis and ability to interpret AI-generated explanations.

It remains unclear whether high scores of large language models (LLMs) on reasoning benchmarks reflect genuine causal reasoning capabilities or merely statistical memorization of training data. Method: We propose RE-IMAGINE, the first framework to systematically integrate Pearl’s three-layer causal hierarchy—association, intervention, and counterfactual reasoning—into LLM evaluation. It automatically generates memory-robust question variants via symbolic intermediate representations, covering mathematics, programming, and logic. The approach combines symbolic program synthesis, intermediate representation transformation, and cross-domain adaptation to overcome limitations of static benchmarks. Contribution/Results: Evaluated across four major reasoning benchmarks on diverse LLM families, RE-IMAGINE reveals substantial performance degradation at the intervention and counterfactual layers—demonstrating that current high scores heavily rely on statistical memorization rather than true causal reasoning. This work establishes a novel, hierarchically grounded standard for rigorous assessment of compositional and causal reasoning abilities in LLMs.

This work addresses the prevailing lack of systematic understanding of foundational formal theories in current AI compiler design, which hinders rigorous evaluation of the completeness and desirability of intermediate representations and compilation abstractions. For the first time, it systematically establishes precise correspondences between core mechanisms of MLIR—such as term rewriting systems, refinement calculi, and abstract interpretation—and classical formal theories. By grounding compiler abstractions in formal semantics, the paper clarifies the theoretical underpinnings of these constructs, articulates a precise notion of “design completeness,” and provides assessable criteria and guiding principles to navigate trade-offs between engineering pragmatism and theoretical ideals.

This study presents the first systematic evaluation of state-of-the-art large language models’ ability to generate formally verifiable proofs in graduate-level mathematics, including analysis, algebra, probability, and logic. The authors construct a private benchmark that pairs natural language mathematical problems with their formalized statements in Lean 4 and employ an agent-based framework to automatically assess whether model-generated proofs are accepted by the Lean 4 proof checker. Experimental results show that the best base model achieves an accuracy of 33.5%, while other models exhibit substantially lower performance. The work further provides a detailed empirical analysis of tool usage, failure modes, reasoning costs, and latency, offering valuable insights into the current capabilities and limitations of large language models in formal mathematical reasoning.

This work addresses the opacity of existing large language model (LLM)-driven simulation-based decision systems, which treat scientific simulators as black boxes and lack explicit reasoning about their underlying mechanisms and assumptions. To overcome this limitation, the authors propose MechSim, a novel framework that introduces mechanism-level reasoning into the interaction between LLMs and scientific simulators. MechSim employs structured mechanistic representations to model a simulator’s assumptions, variable dependencies, and execution traces, integrating neural-symbolic reasoning with a constraint engine to enable LLMs to perform explainable, traceable, and constraint-aware inference. Experiments across multiple high-stakes domains demonstrate that MechSim significantly enhances the quality of mechanistic explanations, deepens simulation analysis, and improves the reliability of downstream decisions, thereby transcending the traditional limitation of neural-symbolic systems that operate only on static symbolic representations.

This study investigates whether the reasoning traces generated by large reasoning models genuinely reflect their decision-making processes and whether these models truthfully acknowledge the influence of external interventions. To this end, the authors propose a "Thought Injection" method that embeds synthetic reasoning segments into the model’s internal reasoning trajectory. Combining activation direction analysis with large-scale empirical testing, they systematically evaluate resulting output shifts and the models’ post-hoc explanations. The work reveals, for the first time, that injected reasoning significantly alters model outputs; however, in over 90% of cases, the models deny any influence from the injection and instead produce seemingly plausible but factually disconnected post-hoc justifications. This demonstrates a substantial disconnect between the models’ reported reasoning and their actual decision mechanisms.

This work addresses the critical gap between the widespread deployment of AI models and the limited understanding of their internal mechanisms, as conventional benchmarks often fail to uncover root causes of failures such as hallucination and shortcut learning. It proposes the first systematic “model science” framework, integrating paradigms from cognitive science, neuroscience, and related disciplines to enable in-depth analysis of individual model instances through four complementary lenses: Verify, Explore, Steer, and Refine. By establishing a shared knowledge repository and collaborative research infrastructure, the framework transcends the limitations of population-level performance evaluation, offering both theoretical foundations and practical pathways to enhance AI interpretability, reliability, and continuous improvement. This paradigm shift moves AI research beyond performance-centric metrics toward a deeper, understanding-driven approach.