Emergent Hierarchical Monosemantic Neurons from the Group-Contrastive Forward-Forward Algorithm

2026-07-13

AI Summary

Traditional sparse dictionary learning is limited in achieving monosemantic neurons due to non-identifiable objectives, reliance on linear representation assumptions, and insufficient capacity to model nonlinear concepts. This work proposes the Group-Contrastive Forward-Forward (GCFF) algorithm, which dispenses with sparsity assumptions and instead leverages class-specific routing and intra-class contrastive learning under purely feedforward, local learning rules to impose hierarchical structural constraints that encourage the spontaneous emergence of monosemantic neurons. GCFF is the first method to realize hierarchically abstracting monosemantic neurons without global error signals and effectively captures nonlinear concepts. Experiments demonstrate that GCFF successfully recovers monosemantic neurons with increasing abstraction levels on CLIP representations, enabling unsupervised identification of background environmental attributes unrelated to foreground objects; when trained from scratch, it achieves state-of-the-art performance among feedforward algorithms on multiple image classification benchmarks.

Abstract

Mechanistic interpretability has made significant strides in understanding neural network representations, with sparse dictionary learning (SDL) methods, most prominently sparse autoencoders, as a central paradigm. However, recent work has reported several limitations of this paradigm: SDL objectives are non-identifiable; SDL methods rely heavily on the Linear Representation Hypothesis; and a growing body of evidence points to concepts that are encoded non-linearly and are therefore not expressible as any single direction. We hypothesise that a different route to monosemanticity is available. Biological visual systems exhibit highly selective neurons organised into hierarchies of increasing abstraction, and this organisation emerges from local, layer-wise learning rules rather than from a global error signal; we therefore ask whether a biologically plausible learning algorithm will likewise yield monosemantic neurons. To test this, we propose Group-Contrastive Forward-Forward (GCFF), a forward-forward training algorithm that combines class-specific routing with within-class contrastive objectives, reaching monosemanticity through architectural constraints rather than sparsity. Because GCFF attaches multiple non-linear layers to the representation under study, its neurons can therefore capture the non-linear concepts. On CLIP representations, a single trained GCFF module recovers monosemantic neurons whose abstraction increases progressively with depth, reaching environmental properties that hold independently of an image's foreground, without any sparsity constraint or supervision of abstraction level. We further demonstrate that GCFF can train networks from scratch, achieving state-of-the-art performance among forward-forward algorithms on various image classification benchmarks.

Problem

• mechanistic interpretability
• monosemantic neurons
• sparse dictionary learning
• non-linear representations
• hierarchical abstraction

Innovation

• monosemantic neurons
• Group-Contrastive Forward-Forward
• non-linear representation
• hierarchical abstraction
• forward-forward algorithm