Toward a mechanistic understanding of inference in visual cortex and diffusion models

2026-07-17

AI Summary

This work proposes a mechanistic framework to understand perceptual inference in the primary visual cortex (V1) and elucidate the internal workings of diffusion models. We develop a recurrent dynamical system grounded in sparse coding and pairwise interaction priors, trained via denoising score matching and implicit differentiation. Our model establishes, for the first time, a direct link between the horizontal connectivity structure of V1 and the internal mechanisms of diffusion models, offering an interpretable recurrent dynamics framework in which partial latent variables decouple from the input to form hierarchical representations that enforce global consistency. Experiments demonstrate that the model not only reproduces orientation-selective connectivity observed in V1 neurons but also achieves strong denoising performance under high noise levels, with generalization capabilities approaching those of black-box diffusion models while providing a transparent generative mechanism.

Abstract

We describe a model of perceptual inference in primary visual cortex (V1) equivalent to a minimal diffusion model whose function can be readily understood from its parameters. The model is based on sparse coding with a non-factorial prior over latent variables in the form of an unconstrained, pairwise interaction matrix, extending standard sparse coding inference to a general recurrent dynamical system. We efficiently train these recurrent dynamics using a denoising score-matching objective and implicit differentiation. After training on natural images, the learned interaction matrix mirrors the structure of horizontal connections in superficial layers of V1 that link neurons of similar orientation tuning. This model exhibits exceptionally good denoising performance, restoring image features such as extended contours amid extreme visual ambiguity, nearly matching the behavior of standard, black-box diffusion architectures in generalization regime. Owing to the model's simplicity, the network's Jacobian can be decomposed directly in terms of the interaction matrix between latent variables, revealing mechanistically how the recurrent dynamics assign high probability over a continuous family of natural structural deformations. Intriguingly, within this circuit, a large fraction of latent variables learn to disconnect from visual input altogether, essentially forming a hierarchical representation that appears to enforce global consistency among image features. Together, the model and results bridge two distinct domains: for neuroscience, it generates concrete, testable hypotheses regarding functional connectivity in recurrent neural circuits during perceptual inference tasks; for machine learning, it elucidates the internal mechanisms learned by diffusion models that allow them to generate infinitely many novel images from a finite training set.

Problem

• perceptual inference
• visual cortex
• diffusion models
• recurrent dynamics
• mechanistic understanding

Innovation

• diffusion models
• sparse coding
• recurrent dynamics
• perceptual inference
• interaction matrix