IRIS: A Visual Cortex-Inspired Framework for Analyzing Orientation Selectivity in Vision Transformers

2026-08-05

AI Summary

This study investigates how Vision Transformers (ViTs), despite lacking explicit local inductive biases, spontaneously develop low-level visual features such as orientation selectivity during training. Inspired by the biological visual cortex, it introduces for the first time neuroscientific analyses of orientation selectivity into ViT research, proposing novel metrics—including Orientation Recruitment Score (ORS), Optimal Response Similarity (ORS), and tuning bandwidth—to systematically track the emergence and evolution of such selectivity across network depths. The experiments reveal that orientation selectivity is primarily driven by the training objective, strengthening in shallow to intermediate layers during training while giving way to semantic representations in deeper layers. These proposed metrics effectively uncover the underlying encoding mechanisms of ViTs and offer an interpretable basis for selecting optimal layers for efficient fine-tuning on downstream tasks.

Abstract

Vision transformers (ViTs) have become the de facto standard for image encoding across many perception tasks. Despite their empirical success, it remains mechanistically unclear how they encode low-level features, given their lack of inductive biases: ViTs process information globally rather than relying on local structure. Biological visual systems, in contrast, build low-level features, such as orientation selectivity in the primary visual cortex, by combining information from small, localized regions of the visual field. These features are general-purpose representations, shared and required across multiple specialized neural pathways, unlike higher-level, task-specific semantic features. This raises the question if such biologically-grounded features arise in ViTs. In this work, we systematically study how orientation selectivity emerges in ViTs by introducing a suite of neuroscience-inspired metrics: representational similarity score (RSS), orientation recruitment score (ORS), and orientation tuning bandwidth to quantify how orientation is encoded in representational geometry and as a function of model depth. Through extensive analysis, we find that: (1) the training paradigm is the strongest determinant of orientation selectivity, with models sharing an objective, peaking at comparable relative depths regardless of scale (2) many units are orientation-selective early in training, with early-to-middle layers recruiting more such units over time, while deeper layers lose selectivity and broaden their tuning toward semantic encoding and (3) our metrics offer a mechanistic heuristic for how many layers to unfreeze for best downstream generalization. Our framework presents a way to track biologically-grounded features during ViT training, probes how desired properties are encoded in transformer representations, and builds a systematic understanding of how ViTs generalize across tasks.

Problem

• orientation selectivity
• vision transformers
• visual cortex
• inductive biases
• representational geometry

Innovation

• orientation selectivity
• vision transformers
• neuroscience-inspired metrics
• representational geometry
• inductive bias