PDFBench: A Benchmark for De novo Protein Design from Function

2025-05-25

AI Summary

Long-standing limitations in de novo protein design—including the absence of standardized, publicly available benchmark datasets and multidimensional evaluation frameworks—have hindered fair, reproducible comparison across methods. To address this, we introduce PDFBench, the first function-driven comprehensive benchmark supporting both description-guided and keyword-guided design tasks. It features an integrated evaluation framework comprising 22 metrics across five dimensions: sequence validity, structural fidelity, language–protein alignment (a novel dimension introduced herein), novelty, and diversity. We systematically analyze inter-dimensional correlations and provide principled metric selection guidelines. Leveraging sequence modeling, AlphaFold2-based structural validation, natural language understanding, and diversity quantification, we rigorously evaluate five state-of-the-art models, revealing their fundamental capabilities and limitations. All data, code, and evaluation pipelines are fully open-sourced to foster community advancement.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Evaluation and Analysis

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

In recent years, while natural language processing and multimodal learning have seen rapid advancements, the field of de novo protein design has also experienced significant growth. However, most current methods rely on proprietary datasets and evaluation rubrics, making fair comparisons between different approaches challenging. Moreover, these methods often employ evaluation metrics that capture only a subset of the desired properties of designed proteins, lacking a comprehensive assessment framework. To address these, we introduce PDFBench, the first comprehensive benchmark for evaluating de novo protein design from function. PDFBench supports two tasks: description-guided design and keyword-guided design. To ensure fair and multifaceted evaluation, we compile 22 metrics covering sequence plausibility, structural fidelity, and language-protein alignment, along with measures of novelty and diversity. We evaluate five state-of-the-art baselines, revealing their respective strengths and weaknesses across tasks. Finally, we analyze inter-metric correlations, exploring the relationships between four categories of metrics, and offering guidelines for metric selection. PDFBench establishes a unified framework to drive future advances in function-driven de novo protein design.

Problem

• Lack of fair comparison in protein design methods
• Incomplete evaluation metrics for protein properties
• Need for comprehensive benchmark in de novo design

Innovation

• Comprehensive benchmark for protein design
• 22 metrics for multifaceted evaluation
• Supports description and keyword-guided tasks