HELM: Hierarchical Encoding for mRNA Language Modeling

2024-10-16

arXiv.org

AI Summary

Existing language models for mRNA sequence modeling neglect the intrinsic codon-level hierarchical structure and synonymous constraints of the genetic code, resulting in suboptimal biological representational capacity. To address this, we propose a codon-aware pretraining framework: (1) a synonymy-modulated loss function explicitly incorporates codon partitioning and prior knowledge of genetic code synonymy into the training objective; and (2) hierarchical positional encoding captures the trinucleotide periodicity inherent in coding sequences. This work pioneers biologically grounded, mechanism-driven representation learning for language models in genomics. Evaluated on six mRNA property prediction tasks and antibody CDR region annotation, our method achieves an average 8.0% improvement in predictive performance. Moreover, generated sequences exhibit significantly enhanced distributional fidelity and diversity. The framework establishes an interpretable and generalizable paradigm for biomolecular sequence modeling.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Messenger RNA (mRNA) plays a crucial role in protein synthesis, with its codon structure directly impacting biological properties. While Language Models (LMs) have shown promise in analyzing biological sequences, existing approaches fail to account for the hierarchical nature of mRNA's codon structure. We introduce Hierarchical Encoding for mRNA Language Modeling (HELM), a novel pre-training strategy that incorporates codon-level hierarchical structure into language model training. HELM modulates the loss function based on codon synonymity, aligning the model's learning process with the biological reality of mRNA sequences. We evaluate HELM on diverse mRNA datasets and tasks, demonstrating that HELM outperforms standard language model pre-training as well as existing foundation model baselines on six diverse downstream property prediction tasks and an antibody region annotation tasks on average by around 8%. Additionally, HELM enhances the generative capabilities of language model, producing diverse mRNA sequences that better align with the underlying true data distribution compared to non-hierarchical baselines.

Problem

• Incorporates codon-level hierarchical structure into mRNA language modeling.
• Improves mRNA property prediction and antibody region annotation tasks.
• Enhances generative capabilities for diverse mRNA sequence production.

Innovation

• Hierarchical encoding for mRNA codon structure
• Codon-level loss function modulation
• Enhanced generative mRNA sequence capabilities