reporting standards

Current FAIR assessment tools for Open Science Platforms (OSPs) lack systematic, standardized evaluation criteria, hindering comparative analysis and reproducibility. Method: We propose the first horizontal evaluation framework specifically designed for OSP-oriented FAIR assessment tools, conducting a multidimensional comparative analysis of 22 existing tools. Our novel consistency evaluation model assesses principle mapping completeness, automation level, cross-platform reproducibility, and semantic interoperability support. Methodologically, we integrate rule-based validation, API response analysis, metadata parsing, and Delphi expert consensus to quantitatively identify common deficiencies. Results: We uncover 17 recurrent weaknesses—most notably, a 64% gap in assessing the “R” (Reusable) principle. The study delivers an extensible FAIR tool maturity taxonomy, an open-source assessment protocol, and a curated benchmark test suite, substantially enhancing comparability, transparency, and reproducibility of FAIR assessments in practice.

Health sciences—hypothesis-driven and emphasizing reproducibility—clash with visual analytics—iterative, exploratory, and interaction-dependent—leading to cross-disciplinary challenges: terminological misalignment, divergent expectations for data preparation, conflicting validation criteria, and contradictory interpretability requirements. To address this, we propose an integrative framework structured along three dimensions: cultural adaptation, standard harmonization, and process coordination. It specifies seven concrete, actionable steps—the first systematic effort to bridge confirmatory and exploratory research paradigms. Grounded in interdisciplinary co-design, the framework incorporates integrated workflow modeling, a terminology alignment tool, and a multi-stage quality validation benchmark. It enables clinically relevant, reliable, and reproducible collaborative analysis. By fostering deep methodological integration, the framework advances a unified research agenda that enhances scientific rigor, practical feasibility, and clinical translatability of hybrid analytical approaches.

This study addresses the challenge of unreliable AI models and diminished clinical trust stemming from opaque data quality reporting in the secondary use of electronic health records (EHRs). To this end, the authors propose the first comprehensive framework for transparent data quality reporting across the entire EHR lifecycle. The framework innovatively distinguishes between data producers and consumers, explicitly defines five critical phases, and maps established data quality dimensions to specific workflow stages. Through iterative stakeholder and process analysis, a structured reporting mechanism is developed and validated on real-world datasets, demonstrating its ability to effectively trace the origins of data quality issues. The approach significantly enhances data interpretability, fitness-for-use, and governance efficacy, thereby providing a robust foundation for trustworthy AI development and clinical research.

This work addresses the challenge of reproducibility in data visualization scripts, which often lack essential components such as source code, input data, execution environment, or output artifacts. To bridge this gap, we propose yProv4DV—a lightweight Python library that, for the first time, targets script-based visualization workflows by automatically capturing comprehensive provenance information—including source code, input data, runtime environment, and output results—through a single function call. Designed to be minimally invasive and ready-to-use, yProv4DV enables full reproducibility of visualization outputs without requiring any modification to existing scripts. This approach significantly reduces the development burden on researchers striving to ensure reproducibility and fills a critical void in automated provenance support within visualization pipelines.

Scientists frequently record experimental metadata in spreadsheets, yet ensuring consistency and standards compliance remains challenging. This paper introduces a spreadsheet-native metadata governance paradigm: customized Excel/CSV templates embed HuBMAP standards; OWL/SKOS ontology-driven controlled vocabularies are integrated; and a web-based real-time semantic validation tool enables immediate, on-entry verification. The approach seamlessly incorporates semantic constraints into familiar spreadsheet workflows—requiring no platform switching or new system adoption. Deployed across the HuBMAP Consortium, it significantly improved multi-omics metadata compliance rates, increased data entry efficiency, and reduced error identification and correction time by over 70%. To our knowledge, this is the first work to deeply embed ontology-based constraints and real-time semantic validation directly within spreadsheet environments, establishing a scalable, practical paradigm for biomedical metadata standardization.

Design-oriented visualization research often struggles to meet conventional reproducibility standards due to its inherent subjectivity, contextual dependence, and iterative nature, thereby limiting its transparency and rigor. To address this challenge, this work proposes “traceability” as a viable alternative to traditional reproducibility. It presents the first systematic theoretical framework centered on three core components—recording, reporting, and reading—and introduces tRRRacer, a supporting tool implementing this framework. Through collaborative autoethnography, the authors reflect on practical applications of traceability in design-oriented research, demonstrating its feasibility and yielding actionable principles alongside theoretical insights. This approach offers a novel pathway to enhance the rigor and transparency of such studies without relying on strict reproducibility criteria.

This study addresses the lack of systematic understanding regarding the types and motivations of visual representations in qualitative research. Building upon and extending Verdinelli & Scagnoli’s (2013) work through a data-driven literature review, it conducts a content analysis of articles and their visualizations published between 2020 and 2022 in three leading qualitative methods journals. Integrating epistemological stance classification with visualization-type coding, the study innovatively combines correspondence analysis and cognitive network analysis for the first time. Findings indicate that while visualizations remain underutilized in qualitative research, their typological diversity is increasing, and the choice of graphical representation appears largely independent of the authors’ epistemological positions. These results offer both empirical grounding and methodological innovation for integrating interdisciplinary visualization tools into qualitative inquiry.

This study addresses a critical limitation in current learning analytics tools, wherein frequency-oriented visualizations often obscure rare yet educationally significant student feedback. To bridge the gap between quantitative visualization and qualitative educational research, the authors engaged STEM education researchers in analyzing student logs using the WordStream platform. Through an integrated approach combining thematic analysis, member checking, and mixed-methods user research, the study uncovered epistemological tensions educators face when repurposing quantitative codings for qualitative inquiry. Three core themes emerged: tool experience, disciplinary contextualization, and the integration of quantitative and qualitative paradigms. Building on these insights, the work proposes design principles for visualizations that support deep qualitative exploration, offering both theoretical grounding and practical guidance for the next generation of learning analytics tools.

Public sector actors increasingly rely on vendor-provided AI transparency documents, such as FactSheets, for accountability and risk assessment, yet their practical utility remains empirically unexamined. This study addresses this gap through semi-structured interviews and a systematic content analysis of FactSheets published by the GovAI Coalition, revealing for the first time that these documents function dually as both marketing instruments and disclosure mechanisms in practice. The findings indicate that while FactSheets alone are insufficient to support robust technical evaluation, they play a critical role in fostering trust, enabling stakeholder alignment, and sustaining ongoing governance dialogues. Building on these insights, the paper proposes reconceptualizing FactSheets not merely as static informational artifacts but as relational governance tools that facilitate dynamic, iterative engagement between public institutions and AI vendors.

Current AI research tools lack evaluation benchmarks that simultaneously account for usability, interpretability, and integration into scientific workflows, making it difficult to assess their practical reliability in academic settings. This work proposes a comprehensive evaluation framework that integrates human-centered dimensions—such as usability and interpretability—with computational metrics. Through a human-AI collaborative approach—including explainable AI (xAI) analysis, source tracing validation, task-oriented testing, and workflow integration observation—the study systematically evaluates AI-powered question-answering and literature review tools on both exploratory and precision-oriented tasks. Findings reveal a core tension: while these tools effectively support initial exploration by providing useful overviews, they exhibit unreliable precision in factual extraction, with xAI highlights often misaligned with actual answers. Similarly, literature tools aid preliminary discovery but suffer from poor reproducibility and low transparency, necessitating rigorous human verification.