Integrating gene regulatory priors into Transformer attention with scTransformer for interpretable scRNA-seq analysis

2026-06-08

AI Summary

Existing Transformer-based methods for single-cell transcriptomic analysis overlook gene regulatory relationships, limiting model interpretability and robustness. This work proposes scTransformer, which, for the first time, explicitly integrates prior knowledge of gene regulatory networks into the Transformer’s attention mechanism by constraining information flow to align with known regulatory structures, thereby learning biologically meaningful cell representations. Evaluated under both self-supervised and supervised training frameworks on single-nucleus RNA-seq data, scTransformer significantly outperforms standard Transformers: it achieves higher cell-type classification accuracy, enhances separation of cell types in embedding space, and yields attention patterns that closely recapitulate established regulatory programs. By unifying high performance with strong interpretability, scTransformer advances the development of biologically trustworthy foundation models for single-cell genomics.

Abstract

Motivation: Transformer-based models are increasingly applied to large-scale single-cell transcriptomics, showing strong performance through self-supervised learning on millions of cells. However, most existing approaches treat genes as independent features, and largely ignore prior biological knowledge, which limits interpretability and robustness. In this paper, we explore whether explicitly incorporating gene regulatory information can improve both model performance and biological insight. Results: We present scTransformer, the first Transformer-based approach that builds a priori knowledge of biological mechanisms into the model's attention patterns. By constraining information flow according to known regulatory structures, the model learns representations that are more biologically meaningful. We evaluate scTransformer on a disease-relevant single-nucleus RNA-seq dataset using supervised cell-type classification. Compared to standard Transformers, our approach improves classification accuracy, enhances separation of cell types in embedding space, and produces attention patterns consistent with known regulatory programs. Overall, our results demonstrate that embedding biological structure into Transformer models can enhance interpretability without sacrificing performance, offering a principled step toward biologically grounded foundation models for single-cell omics.

Problem

• gene regulatory priors
• Transformer attention
• scRNA-seq analysis
• interpretability
• biological knowledge integration

Innovation

• scTransformer
• gene regulatory priors
• Transformer attention
• interpretable scRNA-seq
• biologically grounded models