g-DPO: Scalable Preference Optimization for Protein Language Models

2025-10-22

AI Summary

DPO faces scalability bottlenecks in aligning protein language models with experimental design objectives: the number of training preference pairs grows quadratically with sequence count, rendering training prohibitively expensive even on moderate-scale datasets. To address this, we propose Cluster-DPO—the first efficient preference optimization framework tailored to protein sequence space. It reduces redundancy by clustering sequence embeddings to prune non-informative preference pairs and introduces intra-cluster likelihood amortization to substantially lower computational overhead while preserving gradient consistency. Evaluated on three protein engineering tasks—enzyme activity, thermostability, and binding affinity—Cluster-DPO achieves in vitro and in vivo performance comparable to standard DPO, with 1.8–3.7× faster training. Crucially, speedup scales favorably with dataset size. Cluster-DPO thus establishes a scalable new paradigm for aligning large-scale protein language models with experimental objectives.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Learning Preferences or Rankings
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Direct Preference Optimization (DPO) is an effective approach for aligning protein language models with experimental design goals. However, DPO faces a scalability bottleneck: the number of possible training pairs grows quadratically with the number of labeled sequences, leading to prohibitive training times even for modestly sized datasets. We introduce g-DPO, a framework that (i) uses sequence space clustering to prune redundant pairs while preserving training signal, and (ii) amortizes likelihood computations with group-based approximations. Across three protein engineering tasks, g-DPO maintains in-silico and in-vitro performance that is statistically indistinguishable from standard DPO, while converging 1.8 to 3.7 times faster, with greater gains expected as the size of the dataset increases.

Problem

• Optimizing protein language models with scalable preference learning
• Reducing quadratic training pair growth in DPO methods
• Accelerating convergence while maintaining experimental performance standards

Innovation

• Uses sequence space clustering to prune redundant pairs
• Amortizes likelihood computations with group approximations
• Maintains performance while converging significantly faster