technical documentation

Current research software publishing lacks automated tools to ensure FAIR (Findable, Accessible, Interoperable, Reusable) compliance. To address this, we propose HERMES—a CI-based, extensible software publishing workflow that automatically generates software artifacts enriched with persistent identifiers (PIDs) and structured metadata. Our contributions are threefold: (1) the first configurable, reusable publishing pipeline designed for the full lifecycle of research software; (2) a modular plugin architecture enabling flexible integration of heterogeneous metadata sources; and (3) a Python-driven CLI toolchain with native GitHub Actions support. Evaluated across three cross-domain empirical case studies, HERMES significantly reduces manual publishing effort, improves metadata completeness by up to 40%, and increases FAIR compliance rates by over 65%. The workflow provides foundational infrastructure for sustainable, standards-compliant research software publication.

Real-world API documentation is often unstructured and highly heterogeneous across tools, leading to high development costs and poor generalization in agent construction. Method: This paper proposes an end-to-end, scalable tool generation framework: (1) parsing raw API documentation to automatically extract interface semantics and parameter constraints; (2) generating executable Python tool functions; and (3) incorporating a code-agent-driven iterative feedback optimization mechanism to improve tool invocation accuracy. Contribution/Results: To our knowledge, this is the first approach achieving high validation rates (>92%) in fully automated tool generation directly from real-world API documentation—without human annotation or domain-specific adaptation. On the WebArena benchmark, it improves task success rate by 55% and reduces tool construction cost by 90%. Furthermore, it demonstrates strong cross-domain generalization, validated in complex vertical domains such as sugar materials science.

This work proposes ReproScore, a novel framework that explicitly decouples reproducibility readiness (RRS) from reproducibility outcomes (ROS)—a distinction often conflated in existing tools that mistakenly treat static repository completeness as a proxy for successful execution. RRS comprises 26 fine-grained submetrics assessing code and documentation quality, while ROS is derived through sandboxed execution probes. The two dimensions are integrated into an adaptive composite score (RCS), with customizable metric weights via versioned YAML configurations contributed by the community. Empirical evaluation across 423 GitHub repositories spanning five failure modes demonstrates that environment-related metrics effectively discriminate failure types, yet reveals a near-zero correlation between RRS and actual execution success—exposing a critical “readiness–outcome gap” and thereby validating the necessity and efficacy of the proposed architectural separation.

Researchers face challenges in academic writing—including verbose LaTeX coding, poor version control, low collaborative efficiency, and irreproducible results. To address these, this paper proposes a GitHub-based automated manuscript generation framework. It adopts Markdown as the source format and integrates automated LaTeX compilation, programmatic figure generation, and continuous integration (CI) pipelines to enable end-to-end, traceable, and reproducible scientific writing—from raw data to publication-ready PDF. Crucially, the framework redefines manuscripts as “executable outputs,” unifying version control, computational provenance tracking, and dynamic content updating. Evaluated in computational biology and microscopic image analysis workflows, the framework significantly improves collaborative productivity and adherence to open science principles, while supporting high-quality, fully reproducible scholarly publishing.

Official API documentation often fails to meet developers’ needs due to being outdated and incomplete. This work proposes an automated approach that, for the first time, integrates fine-grained API knowledge extraction, dense retrieval, and large language model–based summarization to generate structured documentation from community content such as Stack Overflow. By fine-tuning a dense retrieval model to identify seven categories of API knowledge and incorporating hallucination mitigation and redundancy reduction mechanisms, the method significantly enhances generation quality. Experimental results show that it improves accuracy by up to 77.7% over baseline methods, reduces redundant content by 9.5%, and recovers 34.4% of critical knowledge missing from official documentation. User studies further confirm its substantial advantages in comprehensiveness, conciseness, and practical utility.

This work addresses the widespread irreproducibility of academic Jupyter Notebooks caused by environment drift, missing dependencies, and implicit execution assumptions. The authors propose the first web-oriented, automated reproducibility engineering pipeline that systematically reconstructs and evaluates repository-level execution environments for notebooks hosted on GitHub. By leveraging dependency inference, auto-generated Docker containers, and isolated execution, the pipeline enables large-scale assessment of reproducibility. A novel four-category execution outcome framework is introduced to quantify reproduction fidelity. Evaluation on 443 real-world notebooks shows that containerization resolves 66.7% of dependency-related failures; however, only 46.3% achieve high output fidelity, demonstrating that while containerization is necessary, it is insufficient for bit-for-bit reproducibility. These findings underscore the critical need for systematic reproducibility evaluation in computational research.

This work addresses the challenges of manual Web API integration testing, which is time-consuming, error-prone, and often misaligned with business requirements. The authors propose a novel approach that synergistically combines large language models (LLMs), retrieval-augmented generation (RAG), and prompt engineering to jointly parse natural language business requirements and OpenAPI specifications, thereby automatically generating executable test scripts that are both semantically meaningful and syntactically correct. Evaluated on ten real-world APIs, the method successfully produced valid tests for 89% of the business requirements within three attempts, uncovered multiple previously unknown integration defects, and substantially reduced the manual effort required for test development.

While existing OpenAPI specifications are structurally compliant, they often fail to meet the semantic requirements of AI agents for effective task planning and API invocation, leading to systematic failures. This work proposes Hermes, a multi-agent large language model system that pioneers the application of multi-agent LLMs to endpoint-level semantic readiness assessment of large-scale, production-grade APIs. Hermes analyzes OpenAPI specifications, identifies RESTful and documentation-related code smells, and generates interpretable diagnostic reports. Evaluated on 16 real-world APIs encompassing approximately 600 endpoints, Hermes detected 2,450 such smells, with high precision confirmed by developer validation. The findings have prompted enterprises to incorporate semantic documentation quality into their API governance frameworks and to redefine API design standards for AI agent compatibility.

This study addresses the lack of systematic understanding regarding the implementation and maintenance of the Model Context Protocol (MCP) in real-world open-source projects. To bridge this gap, we introduce a transparent, reproducible multi-stage validation pipeline that integrates GitHub REST/GraphQL APIs with custom Python scripts to systematically annotate structural evidence, classify repository roles, and filter out non-functional examples from 3,238 candidate repositories. This process yields a high-quality dataset of 2,297 verified MCP projects, achieving a validation precision of 83% at 95% confidence. Our analysis reveals Python and TypeScript as the dominant implementation languages and identifies hybrid architecture as the most prevalent design pattern, thereby establishing the first large-scale empirical benchmark for MCP ecosystem research.

This work addresses the high cost, labor intensity, and poor reproducibility inherent in manual evaluation of software engineering artifacts. To this end, we propose Artisan, an intelligent agent powered by large language models that formalizes research reproduction as a standalone code generation task for the first time. Artisan incorporates a novel automated evaluation mechanism that deliberately withholds ground-truth results to guide the agent toward accurate reproductions without leakage. We also introduce Artisan-Bench, the first benchmark specifically designed for automated assessment in software engineering. Experimental results demonstrate that Artisan successfully generates 44 correct reproduction scripts out of 60 tasks, achieving 3.14× the performance of baseline methods at an average cost of 0.45 hours per task, and uncovers 20 previously unknown errors in published papers or their associated artifacts.