BarcodeMAE+: Rethinking Masked Pretraining and Global Representations for DNA Barcode Foundation Models

2026-09-26

AI Summary

This study addresses the pretraining-inference mismatch and the absence of global representations in masked pretraining for DNA foundation models. To overcome these limitations, we propose BarcodeMAE+, which optimizes the encoder-decoder architecture by introducing an explicit [CLS] token for global representation and a region-dependent auxiliary loss function, effectively bridging the pretraining-inference gap. This work establishes the advantages of the MAE-LM architecture and reveals the critical value of [CLS] representations. Evaluated on the BIOSCAN-5M and UNITE datasets, the proposed model achieves a classification accuracy of 80.65%, significantly outperforming existing baseline methods. These results demonstrate that BarcodeMAE+ provides a superior paradigm for DNA barcode modeling.

Abstract

Many DNA foundation models are pretrained by masking parts of a sequence and asking the model to reconstruct them. Standard masked pretraining exposes the encoder to special [MASK] tokens that are absent at inference, creating a mismatch between training and downstream use. The role of an explicit global sequence representation such as a [CLS] token and how it should be trained also remain poorly understood for DNA barcodes. We introduce BarcodeMAE+ and study model architecture, global [CLS] representation, and auxiliary pretraining objectives across arthropod COI (BIOSCAN-5M) and fungal ITS (UNITE+INSD) barcodes. Across both barcode regions, the encoder-decoder MAE-LM architecture outperforms its matched encoder-only counterpart in nearly all evaluated configurations, supporting MAE-LM as an effective architectural design for DNA barcode foundation models. A trained global [CLS] representation provides substantial additional gains: on BIOSCAN-5M, [CLS] accuracy increases from 47.53% without an auxiliary objective to 80.65% with cross-entropy genus classification. The best auxiliary objective is region-dependent: cross-entropy performs best on BIOSCAN-5M, whereas pairwise same-genus classification performs best on UNITE+INSD, reaching 73.19% on Yeast and 63.07% on Filamentous Fungi. BarcodeMAE+ outperforms published DNA foundation model baselines on BIOSCAN-5M and achieves the highest Yeast accuracy among the evaluated UNITE+INSD baselines using frozen encoder representations. Similarity-weighted softmax KNN voting further stabilizes accuracy as neighbourhood size increases. Overall, encoder-decoder masked pretraining and an explicitly trained global representation are strong design choices for DNA barcode foundation models, while the optimal objective for learning that representation depends on the biological domain.

Problem

• DNA barcode foundation models
• masked pretraining
• global representation
• training-inference mismatch
• [CLS] token

Innovation

• Masked Autoencoding
• DNA Barcode Foundation Model
• Global Representation
• Auxiliary Pretraining Objective
• Encoder-Decoder Architecture