RAGenome: Scaling Retrieval-Based Genomic Language Models to Long Contexts

2026-10-08

AI Summary

This study addresses the high computational costs and limited context windows of existing genomic language models, which hinder their ability to capture cross-species evolutionary dynamics and long-range intra-species interactions. We propose RAGenome, the first retrieval-augmented large genomic model, featuring a novel retrieval architecture based on multiple sequence alignment (MSA). By integrating whole-genome alignments across hundreds of vertebrate species with retrieval-augmented generation (RAG) techniques, RAGenome extends the pretraining context by two orders of magnitude, enabling unified, cost-effective modeling of evolutionary relationships and long-range interactions. Under minimal computational resources, the model improves gene discovery performance from 0.45 to 0.60 while maintaining competitive pathogenic variant prioritization. Ultimately, this efficient architecture overcomes short-context bottlenecks to achieve state-of-the-art performance in genomic analysis.

Abstract

The genome holds the blueprint that governs the biological properties of the cell. Consequently, advancing our knowledge of genomic function is crucial both for a broader understanding of biology and for continued biomedical advances. The success of large language models on natural language and protein sequences has motivated similar efforts on genomic data. However, standard genomic language models (gLMs) often require extremely large computational resources and still fall behind traditional methods on some downstream tasks. Recently, MSA-based pretraining has been proposed as an efficient alternative, but existing models are limited to short input contexts, restricting their use to short-range tasks, such as variant effect prediction. In this work, we present RAGenome, the first retrieval-based gLM that scales pretraining to longer contexts (100$\times$ longer than existing MSA-based gLMs), allowing it to capture both across-species evolutionary relationships and within-species longer-range interactions. Trained on whole-genome alignments from 100 vertebrates, RAGenome substantially improves the long-range capabilities of MSA-based gLMs, raising gene finding performance from 0.45 to 0.60, while remaining competitive on purely evolutionary-based tasks like prioritizing pathogenic variants. RAGenome provides competitive gLM performance at a fraction of the training cost, unifying evolutionary modeling and long-range capabilities within a single, flexible, scalable framework. Code is available at https://github.com/PanosAntoniadis/RAGenome.

Problem

• Genomic Language Models
• Long Contexts
• Multi-Sequence Alignment
• Computational Resources
• Long-range Interactions

Innovation

• Retrieval-Based Genomic Language Model
• Long Context Scaling
• Multiple Sequence Alignment
• Whole-Genome Alignments
• Evolutionary Modeling