experimental results analysis

This study addresses a critical limitation in traditional reproducible research, where sharing only code and results fails to expose the implicit assumptions, expectations, and premises underlying an analyst’s reasoning—thereby hindering thorough evaluation of analytical quality. To overcome this, the paper proposes a formal modeling framework that explicitly translates the analyst’s tacit reasoning process into structured logical representations, statically capturing the construction logic of the analysis. This approach enables systematic scrutiny of the analytical chain of reasoning, assumption sensitivity, and conclusion robustness—even in the absence of the original data. Empirical validation on representative data analysis tasks demonstrates the framework’s effectiveness, achieving both logical visualization and data-free static assessment of analytical integrity.

Design-oriented visualization research often struggles to meet conventional reproducibility standards due to its inherent subjectivity, contextual dependence, and iterative nature, thereby limiting its transparency and rigor. To address this challenge, this work proposes “traceability” as a viable alternative to traditional reproducibility. It presents the first systematic theoretical framework centered on three core components—recording, reporting, and reading—and introduces tRRRacer, a supporting tool implementing this framework. Through collaborative autoethnography, the authors reflect on practical applications of traceability in design-oriented research, demonstrating its feasibility and yielding actionable principles alongside theoretical insights. This approach offers a novel pathway to enhance the rigor and transparency of such studies without relying on strict reproducibility criteria.

Scientific computing artifacts—such as analysis scripts and software prototypes—frequently suffer from poor reproducibility due to environmental heterogeneity, dependency drift, and inadequate documentation, thereby undermining research credibility. To address this, we introduce the first cross-disciplinary, structured, and standardized benchmark dataset for computational experiments, encompassing workflows ranging from single-script executions to multi-language, complex pipelines. Our framework uniformly models metadata, standardizes dependency declarations (e.g., requirements.txt, Dockerfiles), encapsulates multi-language execution procedures, and prescribes a rigorous documentation protocol. The dataset comprises dozens of human-validated, fully reproducible experimental cases, enabling objective, comparable, and reproducible evaluation of reproducibility tools. This work fills a critical gap in the field by providing the first systematic, community-grounded benchmark for assessing computational reproducibility, thereby significantly enhancing the rigor, transparency, and comparability of reproducibility research.

This work proposes an AI agent–driven workflow to address the high costs of reproducing large-scale empirical studies, which often stem from discrepancies in computational environments, code, and documentation. The approach decouples scientific reasoning from computational execution: researchers supply standardized diagnostic templates, and the system automatically retrieves and orchestrates reproduction materials within a version-controlled environment. A structured knowledge layer captures failure patterns, enabling adaptive reproduction across heterogeneous studies while ensuring transparency and stability of the analytical pipeline. Evaluated on 92 instrumental variable studies, the method achieves an 87% end-to-end reproduction success rate; when data and code are available, it attains 100% success at both the paper and model levels.

Experimental reproducibility in Empirical Software Engineering (ESE) is hindered by a fundamental disconnect between idealized methodological assumptions—e.g., standardized protocols and controlled conditions—and researchers’ actual experimental practices. Method: We conducted a two-year ethnographic study involving participant observation, in-depth interviews, and content analysis of experimental artifacts across diverse ESE research teams. Contribution/Results: We identify four critical dimensions—activity diversity, role distribution, conceptual granularity, and domain perspective—in which real-world experimentation systematically deviates from textbook models. Based on these findings, we propose the first high-fidelity conceptual and process model grounded in empirical research practice, explicitly capturing the “practice gap” underlying irreproducibility. This model provides foundational evidence and design principles for developing next-generation reproducibility-support tools, methodological guidelines, and evaluation frameworks in ESE.

This study addresses the challenge of verifying defects in AI research—particularly errors relying on prior knowledge—when only outputs are available. To this end, it introduces a novel "research contract" mechanism that binds experimental choices, execution obligations, and evidence, thereby establishing verification boundaries under information asymmetry. The proposed approach enables automated verification through deterministic checkers, registration of fault-specific rules, and metadata filtering, while explicitly distinguishing contract-relative verification from scientific truth. Experimental results demonstrate that the system successfully detected all eight registered mutations and identified 104 defects across 144 variant cases, effectively ensuring the reliability of compliance verification.

This study addresses how spatial biologists can guide and validate complex tissue data analysis tasks executed by AI agents. Building upon the Claude Science agent, the authors employ contextual inquiry, formative pilots, and observational experiments to propose four key design directions: execution control, familiar views, source information transparency, and cross-environment accessible verification. The work reveals the epistemic mechanisms through which scientists rely on visual evidence to evaluate AI-generated results. Furthermore, it constructs a comprehensive empirical model of the analytical workflow encompassing both interactive control and verification. Ultimately, this research contributes a systematic design framework for human-AI collaborative scientific discovery, offering actionable insights into integrating intelligent agents within rigorous biological research practices.

This study addresses the substantial impact of reviewer subjectivity on the reliability of scholarly evaluations. Leveraging 239,521 peer review records from the H1 Connect platform, the authors employ multilevel linear modeling and variance decomposition to systematically quantify the dominant contribution of reviewer-related variability for the first time. The results reveal that reviewer-level effects account for 61% of the total variance in scores—far exceeding the combined influence of manuscript and journal factors (7%) and demographic or institutional biases such as author gender or affiliation (<1%). These findings demonstrate that “reviewer noise” constitutes the primary source of evaluation bias, prompting the authors to advocate for the implementation of “noise audits” in high-stakes academic assessments to enhance fairness and scientific rigor.

This study addresses the lack of executable and verifiable knowledge representations in existing meta-analyses, which hinders the traceability and reproducibility of critical analytical decisions. To overcome this limitation, the authors propose Executable Analytical Knowledge Representation (EAKR) and introduce MetaSynDec, an agent-based framework that, for the first time, enables explicit modeling, machine-actionable execution, and closed-loop validation of meta-analytic decisions. The system leverages large language models to generate structured knowledge and validates and executes it through deterministic, schema- and contract-based services. Evaluated across 58 synthesis units, EAKR successfully constructed all units, achieved exact evidence-set consistency in 75% of cases, and produced confidence intervals overlapping with published results in 98.2% of cases—substantially outperforming direct LLM-generated approaches.

本文提出REPVIS2设计空间，通过八个维度描述复制研究与参考研究的关系，以解决复制研究设计难以描述和比较的问题。