data deduplication

This study addresses the challenge of detecting unlabeled cross-record media duplicates in cultural heritage databases by formulating it as a positive-unlabeled (PU) learning task. The authors propose a lightweight method that requires no explicit negative samples: a query encoder is trained using only a single positive example, leveraging data augmentation and latent representation learning, while L2-norm thresholds in the embedding space transparently filter candidate clones. This approach represents the first effective application of PU learning to media deduplication in cultural heritage contexts. Evaluated on the AtticPOT dataset, it achieves an F1 score of 90.79 (AUROC = 98.99), outperforming the SVDD baseline by 7.70 F1 points; on CIFAR-10, it attains an F1 of 96.37, demonstrating robustness to viewpoint and condition variations. The method is well-suited for curator workflows involving duplicate detection and record linkage.

As compute capacity grows, the scalability of high-quality pretraining data has become a critical bottleneck. Method: This paper systematically investigates the modeling efficacy of small, filtered, and deduplicated datasets under varying compute budgets. We propose a novel document-level differential resampling paradigm: (i) dynamically assigning repetition counts per document based on quality scores; (ii) jointly optimizing learning rate, batch size, and training epochs; and (iii) applying token-budget-constrained data reweighting. Contribution/Results: Experiments reveal— for the first time—that training a strongly filtered dataset for 10 epochs significantly outperforms single-epoch training on an unfiltered dataset ten times larger in size. Under identical token budgets, our approach substantially improves model performance, empirically validating that non-uniform data quality fundamentally alters scaling laws. This work establishes a scalable, quality-centric paradigm for efficient utilization of compact, high-fidelity pretraining corpora.

This study addresses the challenges of many-to-many matching, data ambiguity, and inconsistency arising from heterogeneous cross-cultural institutional records in art history and photographic research. Taking the PHAROS consortium as a case study, it proposes an entity alignment framework that explicitly models uncertainty by integrating institutional cataloging traditions and data granularity. Through case analyses, workflow evaluations, and semantic modeling, the approach incorporates strategies such as anonymous entities, umbrella terms, and uncertainty qualifiers to effectively manage complex alignment scenarios. The research reveals the prevalence of non-one-to-one relationships among cultural heritage entities and distills practical alignment guidelines tailored to the domain. These contributions provide both theoretical grounding and actionable methodologies for cross-institutional data integration, supporting the development of more robust, interoperable, and sustainable cultural heritage infrastructures.

Enterprise-scale entity resolution (ER) faces dual challenges of scalability and accuracy under high-concurrency, massive-data conditions. This paper proposes MERAI—an AI-powered, end-to-end entity matching pipeline integrating adaptive blocking, hardware-aware inverted indexing, and a lightweight semantic similarity model—to jointly optimize precision and memory efficiency. Compared to state-of-the-art systems Dedupe and Splink, MERAI achieves up to an 8.2% F1-score improvement and 3.1× higher throughput on datasets containing up to 15.7 million records, while demonstrating strong robustness and near-linear scalability. To our knowledge, this is the first work to co-design learnable blocking strategies with hardware-optimized indexing for large-scale ER. MERAI establishes a new paradigm for industrial data integration—delivering high accuracy, computational efficiency, deployment readiness, and operational stability in production environments.

To address low OCR accuracy caused by degradation in historical document images, this paper proposes a two-stage end-to-end optimization framework. In the first stage, a U-Net–based image restoration model is trained on a synthetically generated multi-degradation dataset to jointly optimize visual clarity and linguistic consistency. In the second stage, a semantic-aware ByT5 model performs post-OCR error correction, enhanced by a multi-directional block extraction and fusion mechanism tailored for large-format documents. The key innovations include the first joint optimization of image restoration quality and text semantic consistency, and the construction of the first cross-lingual (English/French/Spanish) synthetic dataset for historical text. Evaluated on 13,831 pages of real historical documents, the framework reduces character error rate by 63.9–70.3% over baseline OCR systems, demonstrating substantial improvement.

Weak schema constraints in knowledge graphs often lead to predicate redundancy, resulting in semantic duplication, hindered reuse, and degraded data quality. This work is the first to frame predicate redundancy as a core data quality issue and proposes a closed-loop governance framework encompassing detection, resolution, and prevention. The approach integrates automated techniques—such as embedding-based clustering—with human-in-the-loop validation and embeds this synergy into a crowdsourced knowledge graph evolution pipeline. By extending the SciKGDash platform with interactive review capabilities and support for predicate merging or deletion, the system enables semi-automated curation. Evaluation on ORKG reveals that up to 30% of predicates are redundant, primarily due to user behavior and interface design flaws, thereby demonstrating the effectiveness of the proposed human–machine collaborative strategy.

This work addresses the poor generalizability of existing self-service entity resolution methods on unseen datasets and their difficulty in balancing precision and recall, which often leads to error propagation and cascading erroneous merges. To overcome these limitations, the authors propose a robust self-service entity resolution system that integrates three key innovations: automated selection among multiple algorithms, decoupled optimization of precision and recall—enforcing precision through rule-based veto mechanisms while enhancing recall via diverse candidate generation—and non-transitive cross-cluster merge validation to prevent error propagation. Extensive experiments on six real-world benchmark datasets, ranging from 864 to 5 million records, demonstrate that the proposed approach significantly reduces incorrect merges, enhances practical utility, and substantially lowers the trial-and-error burden for practitioners.

This study addresses the limitations of historical document digitization, which are often hindered by high OCR noise and rigid keyword-based retrieval, leading to suboptimal search performance. The work proposes the first end-to-end framework for historical archive processing and semantic retrieval, uniquely integrating large language models (LLMs) throughout the entire pipeline. Specifically, LLMs are employed to refine OCR output, while semantic vector retrieval, cross-encoder reranking, and retrieval-augmented generation (RAG) enable natural language question–answer–driven interactive search. Evaluated on a corpus of 500,000 Swiss historical newspaper pages, the approach reduces OCR character and word error rates by 44.52% and 60.95%, respectively, and achieves a 31.9% improvement in NDCG@10 over keyword-based baselines, significantly enhancing answer accuracy and contextual relevance.

This work addresses the challenge of efficiently and accurately detecting whether a given text is partially or fully contained—particularly via near verbatim copying—within massive web-scale corpora. To this end, the authors introduce FindMyText, an open-source tool that leverages document fingerprinting with a novel mechanism for identifying contiguous matching fingerprint sequences, explicitly capturing near-exact copied segments rather than relying on holistic text similarity. The system incorporates a distributed disk-based index to enable scalable processing. The study also establishes the first benchmark specifically designed for text containment tasks, demonstrating that FindMyText significantly outperforms existing methods across diverse datasets including arXiv, Wikipedia, and general web corpora, thereby validating its efficiency, robustness, and practical utility.

This work addresses the scarcity of large-scale, real-world multimodal datasets integrating tables, text, and images by introducing ArtiFact—the first multimodal benchmark dataset tailored to the cultural heritage domain. ArtiFact comprises 651,045 structured records, textual descriptions, and images sourced from three major museums, and supports two core tasks: cross-modal error detection and semantic querying. The authors define seven fine-grained error categories and establish a comprehensive framework encompassing multi-source data collection, cross-modal alignment, synthetic error injection, and semantic query evaluation. Experimental results demonstrate that current methods struggle to reliably detect subtle inconsistencies—such as material or period mismatches—and perform inadequately on complex semantic queries, thereby highlighting ArtiFact’s value and challenge as a new benchmark for multimodal data management.