Causal dictionary learning reveals and validates transcription-factor binding features in genomic language models

2026-07-21

AI Summary

This study addresses the limited interpretability of internal representations in genomic language models, which often conflate genuine biological signals with sequence artifacts. The authors propose a novel approach combining sparse dictionary learning with causal interventions to extract interpretable features from the hidden activations of Nucleotide Transformer and DNABERT-2. To mitigate confounding effects from GC content and repetitive elements, they introduce a composition-matched, site-resolved validation protocol. Through directional ablation experiments, they establish—for the first time—the causal role of specific model features in mediating cell type–specific transcription factor binding (CTCF, GATA1, REST) during forward propagation. The method consistently recovers 7–14 causal features per factor, with no signal detected in negative controls, demonstrating both reliability and reproducibility.

Abstract

Genomic language models achieve strong performance across regulatory-genomics tasks, yet what these models internally represent remains opaque, and the field lacks a principled procedure for verifying that an apparent ``concept'' inside a model is real rather than an artifact of sequence composition. We introduce a framework that combines sparse dictionary learning with causal intervention to extract, validate, and causally test interpretable features in genomic foundation models. Training top-$k$ sparse autoencoders on the hidden activations of two architecturally distinct models, Nucleotide Transformer ($6$-mer tokenization) and DNABERT-2 (byte-pair encoding), we recover thousands of monosemantic features that map to transcription-factor (TF) sequence motifs. We show that the naive validation of such features against position weight matrices is severely confounded by GC composition and repetitive elements, producing hundreds of spurious ``TF features'', and we develop a composition-matched, binding-resolved protocol that removes these confounds. Critically, we move beyond correlation: by ablating individual dictionary directions during the model's forward pass and measuring the induced shift in the model's own predictive distribution, we establish that specific features are \emph{causally} used to represent cell-type-specific TF binding, not merely motif presence. Across three transcription factors (CTCF, GATA1, REST) and both architectures, causally validated binding features emerge reproducibly ($7$--$14$ of $15$ tested features per condition), while two classes of negative control, scrambled binding labels and randomly selected features, yield no detectable signal. The framework is purely computational, uses only public data, and provides a reusable standard for interpretability claims in genomic deep learning.

Problem

• genomic language models
• transcription-factor binding
• model interpretability
• causal validation
• sequence composition confounding

Innovation

• causal dictionary learning
• genomic language models
• transcription factor binding
• sparse autoencoders
• interpretability