🤖 AI Summary
This study investigates how representational structures of neural language models, machine translation models, and linguistic annotation tasks relate to each other and to human brain fMRI responses. Method: We systematically analyze representational spaces from 100 models using an integrated framework combining encoder-decoder transfer learning, PCA-based dimensionality reduction, cross-task representation alignment, and neurorepresentational mapping. Contribution/Results: We discover that model representations are embedded in a low-dimensional manifold—introducing the novel concept of “language representation embedding”—and establish the first cross-modal mapping between NLP model hierarchies and cortical language processing hierarchies. The low-dimensional structure significantly predicts mapping fidelity from model representations to fMRI responses; critically, the first principal component robustly recapitulates the cortical hierarchy of language processing (R² = 0.73). This work provides a new paradigm for model interpretability and computational neurolinguistics.
📝 Abstract
How related are the representations learned by neural language models, translation models, and language tagging tasks? We answer this question by adapting an encoder-decoder transfer learning method from computer vision to investigate the structure among 100 different feature spaces extracted from hidden representations of various networks trained on language tasks. This method reveals a low-dimensional structure where language models and translation models smoothly interpolate between word embeddings, syntactic and semantic tasks, and future word embeddings. We call this low-dimensional structure a language representation embedding because it encodes the relationships between representations needed to process language for a variety of NLP tasks. We find that this representation embedding can predict how well each individual feature space maps to human brain responses to natural language stimuli recorded using fMRI. Additionally, we find that the principal dimension of this structure can be used to create a metric which highlights the brain's natural language processing hierarchy. This suggests that the embedding captures some part of the brain's natural language representation structure.
