Vision Transformers for Face Recognition Need More Registers

2026-06-10

AI Summary

This work addresses the issue of attention map artifacts in existing concatenation-based patch embedding (CPE) Vision Transformers (ViTs) for face recognition, which, despite outperforming CLS-token approaches, suffer from reduced interpretability due to noisy attention patterns. To mitigate this, the study introduces learnable register tokens into the CPE-based ViT architecture for the first time, jointly feeding them with patch embeddings into a standard ViT encoder. By incorporating only 4–8 register tokens, the proposed method enhances the structural coherence of the attention mechanism. The resulting model, termed ViT-8R, not only yields substantially clearer and more semantically consistent attention maps but also achieves state-of-the-art face recognition accuracy among ViT-based methods on the IJB-B and IJB-C benchmarks.

Abstract

Recent advances in Vision Transformers (ViTs) for face recognition (FR) have moved beyond the standard CLS-token paradigm. In this paradigm, a special classification token (CLS) is prepended to the patch embeddings and used as a representation of the input for downstream tasks. An alternative approach, Concatenated Patch Embeddings (CPE), instead leverages all patch tokens by concatenating them into a single vector, which is then projected into a compact face representation. CPE has been shown to improve recognition performance in comparison to CLS-based ones, but our qualitative analysis of attention maps showed the presence of artifacts that limit their interpretability. To address this issue, we incorporate register tokens, learnable tokens concatenated to the initial patch embeddings, and processed jointly through the ViT encoder blocks. This mechanism has been shown to produce more structured and interpretable attention maps compared to baseline ViT. We empirically demonstrate that these artifacts consistently appear across various ViT backbones, including small and large models, and that introducing register tokens effectively mitigates them. Adding four or eight registers significantly enhances interpretability, with eight registers providing the highest verification accuracies and smoothest attention structures. Our resulting model, ViT-8R, corresponds to a CPE-based ViT-B architecture augmented with eight register tokens achieves state-of-the-art performance among ViT-based FR models on large-scale IJB-B and IJB-C benchmarks. Also, ViT-8R produces substantially clearer attention maps compared with the baseline model, which offer deeper insight into the model's attention behavior (https://github.com/TaharChettaoui/ViT-FR-Registers)

Problem

• Vision Transformers
• Face Recognition
• Attention Artifacts
• Interpretability
• Patch Embeddings

Innovation

• Vision Transformers
• face recognition
• register tokens
• attention interpretability
• concatenated patch embeddings