Scaling an Autoregressive Transformer for Single-Cell Generation

2026-08-03

AI Summary

This study addresses the challenge of self-supervised generation of single-cell gene expression vectors with high biological fidelity. The authors propose a generative model combining a causal Transformer with a learnable quantized VAE tokenizer, pretrained via cross-entropy loss and conditioned on cell type to synthesize new samples that closely approximate the true expression distribution. They report the first discovery of a dual exponential scaling law in single-cell foundation models, which jointly governs performance with respect to model size and data scale, thereby defining a compute-optimal frontier. The model exhibits predictable scaling behavior across varying parameter counts and dataset sizes, achieving high-fidelity generation and providing a strong pretraining foundation for downstream tasks such as perturbation response prediction.

Abstract

We study a self-supervised generation task for single-cell gene expression vectors: given a set of vectors from a cell type, we aim to generate additional gene expression vectors of that cell type. For this task we characterize both the biological fidelity of the generated gene expression vectors and the scaling behavior of the pretraining loss. The model is a causal transformer paired with a learned quantized VAE tokenizer, trained with a cross-entropy loss. To evaluate the model, we condition it on held-out gene expression vectors of a cell type and generate vectors of gene expression, comparing the resulting distribution over gene expression vectors to the ground truth distribution of that cell type. We study the scaling properties of the proposed architecture by varying the number of trained parameters and the amount of training data. To our knowledge, we find the first jointly-fit two-exponent scaling law and compute-optimal frontier for a single-cell foundation model. Finally, we discuss how this pretrained model could be finetuned for perturbation response prediction.

Problem

• single-cell
• gene expression
• autoregressive generation
• scaling law
• foundation model

Innovation

• autoregressive transformer
• quantized VAE tokenizer
• scaling law
• single-cell generation
• compute-optimal frontier