🤖 AI Summary
Cross-platform heterogeneity and incompleteness of metadata for life science software (e.g., bio.tools, Bioconductor) impede reliable software identity resolution.
Method: We propose an automated solution for constructing a FAIR-compliant unified reference repository. This includes the first systematic evaluation of instruction-tuned LLMs (LLaMA, Phi-3) for software entity disambiguation, and introduces a novel multi-model ensemble inference framework with consensus-driven confidence modeling to enhance decision robustness.
Contribution/Results: Our approach achieves >92% precision on real-world cross-source data and releases a human-annotated gold-standard benchmark. The study identifies fundamental limitations of current LLMs in fine-grained semantic discrimination and cross-registry terminology alignment. It provides a reproducible methodology to support sustainable analysis and observatory development for research software.
📝 Abstract
Software is an essential component of research. However, little attention has been paid to it compared with that paid to research data. Recently, there has been an increase in efforts to acknowledge and highlight the importance of software in research activities. Structured metadata from platforms like bio.tools, Bioconductor, and Galaxy ToolShed offers valuable insights into research software in the Life Sciences. Although originally intended to support discovery and integration, this metadata can be repurposed for large-scale analysis of software practices. However, its quality and completeness vary across platforms, reflecting diverse documentation practices. To gain a comprehensive view of software development and sustainability, consolidating this metadata is necessary, but requires robust mechanisms to address its heterogeneity and scale. This article presents an evaluation of instruction-tuned large language models for the task of software metadata identity resolution, a critical step in assembling a cohesive collection of research software. Such a collection is the reference component for the Software Observatory at OpenEBench, a platform that aggregates metadata to monitor the FAIRness of research software in the Life Sciences. We benchmarked multiple models against a human-annotated gold standard, examined their behavior on ambiguous cases, and introduced an agreement-based proxy for high-confidence automated decisions. The proxy achieved high precision and statistical robustness, while also highlighting the limitations of current models and the broader challenges of automating semantic judgment in FAIR-aligned software metadata across registries and repositories.