record linkage

Enterprise-scale entity resolution (ER) faces dual challenges of scalability and accuracy under high-concurrency, massive-data conditions. This paper proposes MERAI—an AI-powered, end-to-end entity matching pipeline integrating adaptive blocking, hardware-aware inverted indexing, and a lightweight semantic similarity model—to jointly optimize precision and memory efficiency. Compared to state-of-the-art systems Dedupe and Splink, MERAI achieves up to an 8.2% F1-score improvement and 3.1× higher throughput on datasets containing up to 15.7 million records, while demonstrating strong robustness and near-linear scalability. To our knowledge, this is the first work to co-design learnable blocking strategies with hardware-optimized indexing for large-scale ER. MERAI establishes a new paradigm for industrial data integration—delivering high accuracy, computational efficiency, deployment readiness, and operational stability in production environments.

To address challenges in multi-source entity resolution (MS-ER)—including poor model reusability under heterogeneous data, high computational overhead in incremental training, blind cross-source model transfer, and low accuracy in threshold-based matching—this paper proposes a lightweight incremental resolution framework grounded in feature distribution similarity. We introduce a Wasserstein-distance-driven model selection mechanism, enabling interpretable and controllable cross-source transfer. Furthermore, we design a retraining-free model reuse paradigm that integrates active labeling with adaptability assessment to ensure incremental stability. Experimental results demonstrate that, at comparable matching quality, our method achieves 48× higher efficiency than state-of-the-art multi-source active learning approaches and 163× higher efficiency than conventional transfer learning methods, significantly reducing both annotation effort and training cost.

This study addresses the entity linking problem in multi-source heterogeneous data lacking unique identifiers. We propose a probabilistic record linkage method that balances accuracy and scalability. Methodologically, we introduce the Stochastic EM algorithm into latent-variable generative models for the first time, explicitly modeling dependencies among link decisions and enforcing one-to-one constraints, while enabling robust linking under variable-quality fields. Our approach innovatively supports dynamic precision–efficiency trade-offs, effectively handling real-world challenges such as information evolution, data entry errors, and low-quality attributes. Extensive evaluation on large-scale real-world healthcare data demonstrates high linkage accuracy; simulation experiments confirm strong robustness to noise and missing values. The open-source R package FlexRL has been released and deployed in production environments.

In heterogeneous data integration, schema matching and entity resolution are significantly affected by domain-specific characteristics, data scale, missingness rates, and attribute overlap—yet existing graph-based methods struggle to jointly leverage structural and semantic information. Method: This paper proposes a context-aware graph embedding framework that unifies tabular structure, column-level textual descriptions, and external knowledge, employing graph neural networks for joint encoding and embedding learning of multi-source heterogeneous data. Contribution/Results: A key innovation is the context-enhancement mechanism, which systematically uncovers how data characteristics influence matching performance and empirically demonstrates that contextual modeling substantially improves robustness and accuracy—especially under challenging conditions such as high missingness rates and prevalent numeric columns. Extensive experiments across multiple domain-specific benchmark datasets show that our method consistently outperforms state-of-the-art graph-based baselines.

Existing methods struggle to align and interpret distribution shifts across heterogeneous, domain-consistent datasets—such as tabular, textual, visual, and time-series data—especially when scale and modality disparities are pronounced, resulting in poor interpretability. This paper introduces the first human-centric, cross-modal distribution discrepancy explanation framework, implemented as an interpretable dataset comparison toolbox. It integrates statistical hypothesis testing, feature importance decomposition, class activation mapping (CAM), contrastive representation learning, and interpretable generative modeling to enable fine-grained, semantically readable attribution and visualization of distributional shifts. Evaluated across diverse real-world scenarios, the framework significantly improves users’ efficiency in understanding shift causes and enhances the accuracy of intervention decisions—thereby overcoming the limitations of conventional black-box shift detection approaches.

This work proposes an unsupervised record linkage method that operates without labeled data by leveraging semantic knowledge from open-source pretrained language models to enable cross-dataset entity matching. Addressing the challenge of simultaneously achieving high accuracy and reliable uncertainty quantification in label-scarce settings, the approach matches or exceeds the performance of supervised methods on multiple benchmarks—including city names, personal names, organizations, and multilingual political party and bibliographic records—despite requiring no human-annotated training data. Notably, it attains supervised-level accuracy under a fully unsupervised setting, supports local deployment without reliance on external APIs, and thereby substantially reduces privacy risks and accessibility barriers. The method completes typical linkage tasks within minutes, demonstrating both efficiency and practicality.

This work addresses the lack of end-to-end public benchmarks for data integration by introducing MaDI-Bench, the first comprehensive benchmark that spans the entire relational table integration pipeline—including schema matching, value normalization, entity blocking, entity matching, and data fusion. To mitigate benchmark saturation, MaDI-Bench incorporates a set of foundational cross-domain tasks along with a mechanism for generating extensible task variants. The benchmark supports both step-wise and end-to-end evaluation of system performance, validated through diverse pipelines ranging from manual and optimal combinations to large language model (LLM)-based approaches. All resources are publicly released to foster reproducible and holistic assessment of data integration systems.

This work addresses the poor generalizability of existing self-service entity resolution methods on unseen datasets and their difficulty in balancing precision and recall, which often leads to error propagation and cascading erroneous merges. To overcome these limitations, the authors propose a robust self-service entity resolution system that integrates three key innovations: automated selection among multiple algorithms, decoupled optimization of precision and recall—enforcing precision through rule-based veto mechanisms while enhancing recall via diverse candidate generation—and non-transitive cross-cluster merge validation to prevent error propagation. Extensive experiments on six real-world benchmark datasets, ranging from 864 to 5 million records, demonstrate that the proposed approach significantly reduces incorrect merges, enhances practical utility, and substantially lowers the trial-and-error burden for practitioners.

This study addresses a critical yet previously unrecognized issue in knowledge graph construction: the coupling between tabular serialization formats and schema constraints, which significantly degrades both factual coverage and graph fidelity—particularly in country-year statistical tables, where it induces entity inflation or extraction failure. The authors formally identify and name this phenomenon “format-constraint coupling,” introduce a direct graph access evaluation paradigm, and release CSVFidelity-Bench, a benchmark comprising diverse table types and gold-standard facts. Through factorial experiments, bootstrap confidence intervals, token ablation studies, and multi-LLM comparisons, they uncover significant positive coupling effects in four out of six datasets (peak effect size +1.180). Direct graph access reveals a quality gap as large as 47.6 percentage points (p<0.0001), substantially exceeding that of standard retrieval-based approaches.

This work addresses the severe fragmentation of open data platforms, which significantly impedes cross-source data discovery and semantic interoperability. To overcome this challenge, the authors propose a unified framework that integrates over 7.6 million datasets through semantic extraction and normalization, a scalable topic-tag graph, multi-entity–enhanced navigation, and a provenance-aware mechanism. The resulting system establishes a context-aware and traceable paradigm for data exploration. Compared to existing platforms such as ChatPD and Google Dataset Search, the proposed approach demonstrates substantial improvements in data coverage breadth, timeliness, and provenance tracking, thereby enabling more efficient and semantically rich cross-domain data discovery.