🤖 AI Summary
Final-layer embeddings from molecular pre-trained encoders are not necessarily optimal for ADMET property prediction; intermediate-layer representations often yield superior performance.
Method: We propose an “evaluate-then-fine-tune” strategy: first, freeze all encoder layers and perform zero-shot evaluation of each layer’s embeddings to identify the optimal representation layer; then, fine-tune only that layer (or its corresponding subnetwork) for downstream tasks.
Contribution/Results: This work is the first to systematically demonstrate the superiority of intermediate-layer molecular representations. We establish a strong correlation between frozen-layer zero-shot performance and fine-tuned accuracy, enabling low-cost, reliable layer selection. On 22 ADMET benchmarks, frozen intermediate-layer embeddings achieve an average improvement of 5.4% (up to 28.6%) over standard final-layer baselines; layer-specific fine-tuning yields an average gain of 8.5% (up to 40.8%), achieving state-of-the-art performance on multiple tasks.
📝 Abstract
Pretrained molecular encoders have become indispensable in computational chemistry for tasks such as property prediction and molecular generation. However, the standard practice of relying solely on final-layer embeddings for downstream tasks may discard valuable information. In this work, we challenge this convention by conducting a comprehensive layer-wise analysis of five diverse molecular encoders across 22 ADMET property prediction tasks. Our results demonstrate that embeddings from intermediate layers consistently outperform final-layer representations. Specifically, using fixed embeddings from the optimal intermediate layers improved downstream performance by an average of 5.4%, reaching gains up to 28.6%. Furthermore, finetuning up to these intermediate layers yielded even greater average improvements of 8.5%, with performance increases as high as 40.8%, achieving new state-of-the-art results on several benchmarks. Additionally, a strong positive correlation between fixed embedding performance and finetuning outcomes supports an efficient evaluate-then-finetune approach, enabling identification of optimal layers with reduced computational cost. These findings highlight the importance of exploring the full representational depth of molecular encoders to achieve substantial performance improvements and computational efficiency. The code is made publicly available at https://github.com/luispintoc/Unlocking-Chemical-Insights.