Identity-Duplication Auditing in National-Scale Neuroimaging Repositories

2026-10-07

AI Summary

This study addresses identity duplication and data leakage arising from identifier conflicts in national neuroimaging repositories by proposing HAPPEN, a human-in-the-loop auditing framework. The method integrates SHA-256 hash fingerprinting with supervised contrastive retrieval to perform end-to-end deduplication audits on T1-weighted brain MRI scans. Furthermore, it introduces a localized deployment strategy enabling cross-institutional transferability without retraining. Evaluated across 95,000 scans, the framework successfully detected over one thousand duplicate groups and fully recovered all genetic reference pairs. These results demonstrate the efficiency, accuracy, and scalability of the proposed workflow for large-scale inexact duplicate identification.

Abstract

National-scale magnetic resonance imaging (MRI) repositories increasingly integrate data from different studies and institutions. However, subject identifiers that are valid only within individual datasets are no longer guaranteed to remain globally unique after aggregation, making it possible for the same subject to be assigned multiple identifiers, which we define as identity duplication. Such duplication can create leakage between training and test data and inflate apparent performance in downstream biomedical studies. Existing methods do not provide an end-to-end, image-based workflow for auditing this problem at repository scale. In this work, we present HAPPEN, a human-in-the-loop pipeline for auditing identity duplication in T1-weighted brain MRI repositories. It combines SHA-256 fingerprinting for exact-duplicate detection with supervised contrastive retrieval of non-identical scans that may originate from the same person. Retrieved pairs are reviewed as candidates in a locally hosted interface rather than automatically classified as duplicates. We deployed the workflow in a 95,129-scan aggregated repository and assessed end-to-end recovery using 54 genetic-reference pairs. Transferability was assessed by locally deploying the same workflow on 22,386 scans at an independent institution without model retraining or image transfer. Deployment in the study repository identified 1,316 exact-duplicate scan groups and 1,275 reviewer-supported near-duplicate subject groups. Of these groups, 56% and 82%, respectively, crossed dataset boundaries. All 54 genetic-reference pairs were recovered. The external team independently completed the full workflow using a locally selected operating threshold and review standard.

Problem

• identity duplication
• neuroimaging repositories
• data leakage
• MRI
• data auditing

Innovation

• Identity-Duplication Auditing
• Human-in-the-loop
• Supervised Contrastive Retrieval
• Neuroimaging Repositories
• Transferability