data harmonization

In the context of open science, there is an urgent need to implement the FAIR principles (Findable, Accessible, Interoperable, Reusable) in scientific data management, yet systematic guidance on achieving FAIR compliance through big data software reference architectures (SRAs) remains lacking. Method: We conducted a rigorous systematic literature review, screening 323 publications—including those from authoritative databases and expert recommendations—and performed structured data extraction and evaluation aligned with predefined research questions. Contribution/Results: The study identifies seven generic FAIR-compliant SRAs, thirteen scenario-specific FAIR pipelines, and three fully FAIR-compatible SRAs. It uncovers critical bottlenecks in metadata standardization, cross-platform interoperability, and long-term reusability. Furthermore, it establishes the first classification framework and empirical evaluation system for FAIR-oriented big data SRAs, thereby filling a significant research gap and providing a methodological foundation and strategic direction for future SRA design, policy formulation, and tool development.

To address the challenge of achieving FAIR interoperability and reusability for multi-source, heterogeneous scientific data—such as the RADx COVID-19 response data—in large-scale environments, this paper proposes a general-purpose, reproducible, and extensible data harmonization framework. Our approach introduces a novel harmonization paradigm based on parameterized primitive operations and automated execution tracing. It integrates a customizable data representation model, a configurable operation library, and mechanisms for transformation logging and dependency tracking—ensuring protocol reproducibility, process auditability, and transformation reusability. Evaluated in real-world deployment within the RADx Data Hub, the framework significantly lowers the barrier to entry for domain experts, improves harmonization efficiency and transparency, and enables high-quality cross-study analyses. This work provides a scalable, transferable technical pathway for FAIR-compliant data integration across diverse scientific domains.

Data heterogeneity—particularly format discrepancies—severely impedes AI model training efficiency and generalization. While existing research predominantly addresses structural or schema-level conflicts, it largely overlooks format-aware data transformation mechanisms. This paper presents the first systematic taxonomy of format-centric data transformation techniques, categorizing them into three classes: AI-input adaptation, format standardization, and structured/unstructured data conversion. Through a comprehensive literature review and comparative analysis, we delineate the applicability and limitations of each strategy. We further propose a transformation-centric data preprocessing framework that emphasizes semantic preservation and enhanced model adaptability. Our work fills a critical gap by providing the first unified survey of modern data transformation methodologies, offering both theoretical foundations and practical guidelines for mitigating the adverse effects of data heterogeneity. (149 words)

This study addresses the challenge of effectively integrating structured data with unstructured text, a longstanding barrier in data management. It presents the first systematic argument for the necessity of textual data integration and introduces a unified framework that synergistically combines natural language processing, knowledge extraction, and traditional data integration techniques. By leveraging semantic alignment, the framework achieves deep integration between textual content and structured schemas, thereby tackling key challenges inherent in heterogeneous data integration. The work comprehensively surveys existing methodologies and outstanding issues, establishing a theoretical foundation for the emerging field of textual data integration and offering clear guidance for future research and practical implementation.

Scientific data often suffers from poor discoverability, limited sharing, inefficient reuse, and high curation costs due to inadequate standardization of fields and terminology. To address these challenges, this paper proposes a FAIR-aligned standardization framework. Methodologically, it integrates structured vocabulary design, context-aware metadata modeling, and data homogenization strategies to systematically mitigate semantic noise and concept explosion. Crucially, it embeds ten actionable, principle-based rules into a dynamic, evolving data governance process. Empirical evaluation demonstrates that the framework significantly improves metadata quality and semantic consistency, reduces data management overhead, and enhances data findability, interoperability, and long-term reusability—thereby enabling robust, real-world implementation of the FAIR principles in scientific research settings.

This study addresses the challenge of collaborative analysis across multi-institutional electronic health record (EHR) systems, hindered by data heterogeneity, semantic inconsistency, and stringent privacy constraints. We propose a unified two-module framework that enables privacy-preserving EHR harmonization across institutions and disparate data models—without sharing raw individual-level data. The framework integrates standardized clinical coding mapping with machine learning–driven representation learning. Accompanied by open-source software and step-by-step implementation tutorials, it supports end-to-end translational research. Empirical validation across multiple real-world healthcare systems demonstrates substantial improvements in data interoperability and reusability, enabling the construction of high-quality, research-ready EHR datasets. The approach exhibits strong generalizability, scalability, and clinical deployability.

This study addresses persistent challenges in the transparency reporting of major social media platforms following the implementation of the Digital Services Act (DSA), including poor data quality, inconsistent formatting, and a lack of interoperability. It presents the first systematic empirical evaluation of DSA-mandated transparency reports from eight leading EU platforms, employing large-scale quantitative analysis and cross-platform structured comparisons across critical dimensions such as content moderation practices, data timeliness, and completeness. The findings reveal widespread issues—including disorganized reporting formats, inconsistent disclosures, and contradictions among enforcement mechanisms—that expose structural gaps in standardized compliance. These results provide crucial empirical evidence to inform improvements in platform accountability and regulatory coordination under the DSA framework.

This study addresses critical challenges faced by regulated enterprises—including cross-system data inconsistencies, reconciliation difficulties, asset record drift, and overreliance on manual audits—by proposing the GERA framework. GERA innovatively integrates deterministic reconciliation, robust anomaly detection based on Z-Score and its variants, governance-driven semantic standardization, and NIST CSF 2.0 security controls within a four-layer architecture comprising ingestion, staging, core modeling, and semantic services. Empirical validation across banking, broadband service providers, and technology firms demonstrates that the framework significantly enhances reconciliation automation and audit readiness, effectively mitigating 39% of compliance deficiencies identified during PCAOB inspections.

This study addresses the lack of systematic evaluation of data quality tools with respect to their measurement capabilities and integration with large language models (LLMs). It presents the first multidimensional assessment framework grounded in real-world enterprise use cases, systematically evaluating six prominent tools—including open-source solutions such as Great Expectations and Deequ, as well as commercial platforms like Informatica and Experian—across dimensions including rule definition, duplicate detection, metric aggregation, and uncertainty handling, along with their LLM integration mechanisms. The findings reveal that commercial tools offer more comprehensive functionality and初步 support for LLM-assisted rule generation, whereas open-source tools provide greater flexibility at the cost of higher implementation effort. Notably, none of the evaluated tools currently enable direct LLM-based data validation. This work provides empirical guidance for selecting data quality tools and advancing their integration with LLMs.

This work addresses the lack of end-to-end public benchmarks for data integration by introducing MaDI-Bench, the first comprehensive benchmark that spans the entire relational table integration pipeline—including schema matching, value normalization, entity blocking, entity matching, and data fusion. To mitigate benchmark saturation, MaDI-Bench incorporates a set of foundational cross-domain tasks along with a mechanism for generating extensible task variants. The benchmark supports both step-wise and end-to-end evaluation of system performance, validated through diverse pipelines ranging from manual and optimal combinations to large language model (LLM)-based approaches. All resources are publicly released to foster reproducible and holistic assessment of data integration systems.

This work addresses the challenge that domain experts face in translating natural language descriptions of data quality requirements into executable analyses, a process often hindered by reliance on data engineers, resulting in inefficiency and high technical barriers. To overcome this, the paper proposes a no-code, model-driven pipeline that leverages a QPM metamodel to define domain-specific quality analysis templates. Coupled with the Constrainify toolchain, it automatically transforms natural language requirements into executable and reusable analytical logic. By integrating model-driven engineering, metamodeling, and no-code web technologies, the approach significantly reduces dependency on technical expertise, enabling efficient, reproducible, and semantically aligned data quality assessments. This advancement enhances both the accessibility and automation of data quality analysis for non-technical domain practitioners.