Modeling Rapid Contextual Learning in the Visual Cortex with Fast-Weight Deep Autoencoder Networks

2025-08-06

AI Summary

How early visual cortex rapidly acquires global image context remains poorly understood, and computational modeling of this mechanism in deep networks is lacking. Method: We propose a hybrid fast-slow weight Vision Transformer (ViT) autoencoder, incorporating Low-Rank Adaptation (LoRA) to instantiate fast weights that emulate short-term memory traces. We complement this with self-attention analysis and manifold transformation to perform functional-level neural circuit simulation. Results: For the first time in ViT architectures, we replicate the neuroscience-observed familiarity effect: familiarity training aligns latent representations in early layers with those in higher layers, substantially enlarges self-attention receptive fields, and enhances manifold structural stability and contextual sensitivity. LoRA amplifies this effect, yielding a computationally tractable framework for modeling rapid cortical contextual learning. Our work establishes a biologically grounded, interpretable paradigm bridging neural computation and transformer-based representation learning.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Representation Learning
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Recent neurophysiological studies have revealed that the early visual cortex can rapidly learn global image context, as evidenced by a sparsification of population responses and a reduction in mean activity when exposed to familiar versus novel image contexts. This phenomenon has been attributed primarily to local recurrent interactions, rather than changes in feedforward or feedback pathways, supported by both empirical findings and circuit-level modeling. Recurrent neural circuits capable of simulating these effects have been shown to reshape the geometry of neural manifolds, enhancing robustness and invariance to irrelevant variations. In this study, we employ a Vision Transformer (ViT)-based autoencoder to investigate, from a functional perspective, how familiarity training can induce sensitivity to global context in the early layers of a deep neural network. We hypothesize that rapid learning operates via fast weights, which encode transient or short-term memory traces, and we explore the use of Low-Rank Adaptation (LoRA) to implement such fast weights within each Transformer layer. Our results show that (1) The proposed ViT-based autoencoder's self-attention circuit performs a manifold transform similar to a neural circuit model of the familiarity effect. (2) Familiarity training aligns latent representations in early layers with those in the top layer that contains global context information. (3) Familiarity training broadens the self-attention scope within the remembered image context. (4) These effects are significantly amplified by LoRA-based fast weights. Together, these findings suggest that familiarity training introduces global sensitivity to earlier layers in a hierarchical network, and that a hybrid fast-and-slow weight architecture may provide a viable computational model for studying rapid global context learning in the brain.

Problem

• Model rapid contextual learning in visual cortex
• Investigate global context sensitivity in neural networks
• Explore fast-weight architectures for brain learning models

Innovation

• ViT-based autoencoder for visual context learning
• LoRA implements fast weights in Transformers
• Fast weights enhance global context sensitivity