The Vendiscope: An Algorithmic Microscope For Data Collections

2025-02-15

AI Summary

Accurately characterizing compositional diversity in high-resolution, large-scale data—such as protein sequences and crystalline materials—remains challenging. Method: This paper introduces the “Algorithmic Microscope” paradigm—the first application of microscopic observation principles to data science—leveraging differentiable Vendi score to quantify the per-point contribution to set-level diversity. It integrates ecology- and quantum-mechanics-inspired diversity metrics, differentiable weighted sampling, and generative model memorization diagnostics. Contributions/Results: Applied to 250 million protein sequences, it identifies over 200 million near-duplicates and reveals a significant performance drop of AlphaFold on high-diversity Gene Ontology (GO) functions. It confirms that >85% of crystals in the Materials Project are near-duplicates. Furthermore, it precisely locates 13 memorized samples across generative models. The work establishes a robust negative correlation between dataset diversity and model performance, providing a novel benchmark for data quality assessment and model robustness analysis.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Search and Optimization: Metareasoning and Metaheuristics
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems

Application Category

• Web Mining and Content Analysis: Web data quality in the era of algorithmically-generated content
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Data transparency and provenance

Abstract

The evolution of microscopy, beginning with its invention in the late 16th century, has continuously enhanced our ability to explore and understand the microscopic world, enabling increasingly detailed observations of structures and phenomena. In parallel, the rise of data-driven science has underscored the need for sophisticated methods to explore and understand the composition of complex data collections. This paper introduces the Vendiscope, the first algorithmic microscope designed to extend traditional microscopy to computational analysis. The Vendiscope leverages the Vendi scores -- a family of differentiable diversity metrics rooted in ecology and quantum mechanics -- and assigns weights to data points based on their contribution to the overall diversity of the collection. These weights enable high-resolution data analysis at scale. We demonstrate this across biology, materials science, and machine learning (ML). We analyzed the $250$ million protein sequences in the protein universe, discovering that over $200$ million are near-duplicates and that AlphaFold fails on proteins with Gene Ontology (GO) functions that contribute most to diversity. Applying the Vendiscope to the Materials Project database led to similar findings: more than $85%$ of the crystals with formation energy data are near-duplicates and ML models perform poorly on materials that enhance diversity. Additionally, the Vendiscope can be used to study phenomena such as memorization in generative models. We used the Vendiscope to identify memorized training samples from $13$ different generative models and found that the best-performing ones often memorize the training samples that contribute least to diversity. Our findings demonstrate that the Vendiscope can serve as a powerful tool for data-driven science.

Problem

• Extend microscopy to computational data analysis
• Identify diversity in large-scale data collections
• Improve ML model performance on diverse data

Innovation

• Algorithmic microscope for data analysis
• Vendi scores measure data diversity
• High-resolution analysis across scientific domains