fidelity evaluation

This study addresses the challenge of evaluating whether open-source language models can effectively serve as proxies to interpret the behavior of closed-source large language models when internal access is unavailable. Employing API-compatible methods—including log-odds probing, leave-one-out attribution, attention analysis, and input ablation—the authors conduct cross-model comparisons across 11 models spanning four major families: Llama, Qwen, GPT, and Gemini. Their findings reveal that predictive consistency substantially exceeds attribution consistency across model pairs. While white-box signals exhibit stability, they often fail to accurately reflect underlying causal mechanisms; in contrast, black-box input ablation more reliably captures the attribution behavior of closed-source models. These results uncover an “access-effectiveness inversion,” demonstrating that alignment in predictions alone is insufficient to support the transferability of mechanistic interpretations.

This study addresses the limited interpretability of reinforcement learning (RL) policies by proposing an evolutionary optimization-based surrogate fitness framework. The method generates high-informativeness, diverse policy demonstrations via initial-state perturbations and jointly models local behavioral determinism, population-level diversity, and policy optimality—establishing a multidimensional evaluation suite comprising optimality gap, fidelity interquartile mean (IQM), fitness component analysis, and trajectory visualization. It constitutes the first systematic quantification and enhancement of RL policy interpretability. Experiments demonstrate that the framework significantly outperforms random and ablation baselines in discrete Gridworld environments. In continuous control tasks, it provides critical behavioral insights for early-stage policies while enabling high-fidelity refinement of mature policies. The approach thus bridges a key gap between policy performance and human-understandable behavioral rationale, advancing both interpretability assessment and optimization in deep RL.

In high-reliability domains (e.g., healthcare), quantifying the interpretability of reinforcement learning (RL) policies remains challenging due to the lack of objective evaluation criteria and heavy reliance on costly human assessments. Method: We propose the first fully automated, human-free interpretability evaluation paradigm, built upon a simulatability-based empirical framework that integrates program distillation, imitation learning, and symbolic program generation, complemented by computationally tractable interpretability metrics. Contributions/Results: (1) The first scalable, human-free quantitative assessment of RL policy interpretability; (2) Empirical evidence that interpretability and task performance are non-negatively correlated—and synergistically improved in certain settings; (3) Refutation of the existence of a universally optimal policy class across tasks; (4) Strong agreement between automated evaluations and user studies, with all evaluation protocols and baseline code publicly released.

This study reexamines the prevailing assumption in model stealing attacks that high-fidelity surrogate models are functionally equivalent to their targets, through the lens of model multiplicity. By constructing the Rashomon set of a target model and integrating multiplicity-aware metrics—such as ambiguity, disparity, and Rashomon capacity—alongside group fairness measures, the authors systematically evaluate deployment-relevant properties of surrogate models across tabular data, medical imaging, and NLP tasks. Their findings reveal that, despite achieving near-identical prediction fidelity, surrogate models often exhibit substantial discrepancies in critical performance and fairness dimensions, thereby challenging conventional notions of equivalence. This work pioneers the incorporation of model multiplicity into the model stealing literature, uncovering previously overlooked risks associated with deploying high-fidelity surrogates in real-world applications.

To address the model mismatch arising from oversimplified simulation models and the underutilization of empirical measurements, this paper proposes a novel Bayesian surrogate modeling paradigm that jointly leverages simulation and real-world data. Our method introduces a dual-path, multi-source data fusion framework: (1) parallel posterior distribution ensembling and (2) end-to-end joint training, with the first explicit incorporation of empirical data into the Bayesian inference pipeline—endowing simulation models with diagnostic capability. The approach integrates Gaussian process regression, probabilistic distribution fusion, and uncertainty calibration. Evaluated on synthetic and real-world case studies, it achieves an average 23% reduction in RMSE, attains near-theoretical 95% credible interval coverage, and successfully detects structural deficiencies—including missing boundary conditions and omitted physical processes.

This work addresses the challenge of constructing high-accuracy surrogate models in scenarios where high-fidelity data are scarce. We propose a novel multi-fidelity Gaussian process regression method that innovatively embeds low-fidelity data as augmented features into an expanded input space, thereby synergistically combining the strengths of co-kriging and autoregressive modeling. The approach achieves a balanced trade-off between modeling accuracy and computational efficiency within a unified framework, effectively leveraging heterogeneous multi-source data without requiring additional assumptions. Experimental results across multiple benchmark problems demonstrate that the proposed method significantly outperforms existing techniques, delivering higher predictive accuracy at lower computational cost.

While knowledge distillation often preserves task performance in student models, it frequently overlooks degradation in critical capabilities such as uncertainty calibration, boundary behavior, and safety. This work reframes distillation as a lossy projection of teacher behavior and introduces the “Distillation Loss Statement” framework, which integrates context-specific capability preservation objectives to establish a measurable and accountable evaluation paradigm. Through behavioral fidelity analysis, a capability taxonomy, and multidimensional assessment, the study systematically identifies and quantifies non-task-related capability losses. The resulting reproducible taxonomy of distillation-induced losses advances the field beyond mere performance retention toward practical standards that jointly prioritize reliability and responsibility in distilled models.

This study addresses the limited interpretability of large language models (LLMs) in medical prediction tasks, which obscures the medical knowledge encoded within them and potential biases they may harbor. To tackle this issue, the authors propose a prompt-engineering-based surrogate modeling framework that approximates the latent knowledge space of an LLM by generating large-scale input–output pairs, enabling quantitative analysis of the model’s dependence on individual input variables. This approach represents the first application of surrogate modeling to systematically deconstruct the medical knowledge structure embedded in LLMs. The framework successfully identifies erroneous associations contradicting current medical consensus and exposes race-based biases that have been scientifically discredited, thereby offering an interpretable early-warning mechanism to support the safe deployment of LLMs in clinical settings.

High-dimensional stochastic agent-based models (ABMs) are notoriously difficult to analyze systematically due to the curse of dimensionality and inherent stochasticity. This work proposes a multi-stage automated exploration framework that first employs model-driven experimental design to identify key variables and partition the parameter space, then leverages machine learning surrogate models to efficiently capture residual nonlinear interaction effects. The approach operates without human intervention, automatically detecting unstable regions within the simulator and enabling robust sensitivity analysis and policy testing. Applied to a predator–prey case study, the framework successfully isolates dominant variables and highly sensitive nonlinear regimes, substantially enhancing the efficiency and reliability of ABM exploration.

While on-policy distillation (OPD) enhances task performance, it often leads to overconfident and miscalibrated models due to the mismatch between privileged information available during training and the limited observations at deployment. This work proposes CaOPD, a calibration-aware on-policy distillation framework that, for the first time, reveals the decoupling between capability distillation and confidence calibration. CaOPD introduces a student-centric empirical confidence objective, replacing the teacher’s self-reported confidence with empirical confidence estimates derived from student rollouts, thereby achieving a Pareto-optimal balance between calibration and task performance. Experiments demonstrate that CaOPD significantly improves calibration across diverse models and tasks while maintaining competitive accuracy, and exhibits strong robustness in out-of-distribution generalization and continual learning scenarios.