Test-Time Registers as Global Priors for Tokenized Image Generation

2026-07-18

AI Summary

This work addresses the challenge of leveraging registers in Vision Transformers as effective global priors to enhance image generation quality without requiring retraining. The authors propose RegToken, a novel method that repurposes these registers—typically byproducts of attention mechanisms—into plug-and-play global priors. By exploiting their low-frequency concentration during inference, RegToken employs an NFN layer for localization, TokenRank-guided subspace extraction, and a projection-conservation update strategy to compress register information into a small set of prior tokens, which are then integrated into a frozen 1D generative pipeline. Experiments demonstrate consistent performance gains without any training: on ImageNet, the method reduces FID-5k from 20.5 to 20.1, improves the SigLIP score from 3.6 to 3.9, and decreases the required test-time optimization steps from 74 to 52.

Abstract

Attention-based models often develop attention sinks, where a small number of tokens repeatedly attract attention and accumulate unusually large activations. In vision transformers, these outliers are closely related to registers, which have been diagnostically linked to global, low-frequency image structure. Existing work has largely studied registers through interpretability analyses and linear probes, leaving open whether they can be operationalized as plug-and-play signals for generation without retraining. We revisit this question in tokenized image generation. Using OpenCLIP and DINOv2 on ImageNet, we find that test-time register features exhibit stronger low-frequency concentration than both [CLS] readouts and patch-mean features, and show a consistent (albeit moderate) correlation with pixel-space DCT low-frequency energy. Motivated by these diagnostics, we introduce RegToken, a training-free procedure that converts register structure into a small set of global prior tokens by (i) NFN-based layer localization, (ii) TokenRank-guided subspace extraction, and (iii) a projection-and-conservation update on the register subspace. Inserted into a frozen compact 1D token generation pipeline, RegToken improves ImageNet generation and alignment metrics (e.g., FID-5k 20.5 to 20.1, SigLIP 3.6 to 3.9) without modifying pretrained weights, and accelerates test-time optimization (Steps@$τ$ 74 to 52). Overall, our results suggest that structures often viewed as attention artifacts can be repurposed as lightweight global priors for tokenized generation.

Problem

• registers
• tokenized image generation
• global priors
• vision transformers
• test-time

Innovation

• registers
• tokenized image generation
• global priors
• training-free
• RegToken