Cognitive Expert Language Models Better Align with the Corresponding Brain Systems

2026-09-28

AI Summary

This study addresses the limitation of the "one-model-fits-all" paradigm in large language model (LLM)–brain alignment research, which overlooks the functional specificity of brain regions. Through prompt engineering and fine-tuning, we construct expert LLMs tailored to six cognitive domains, including perception and spatial cognition. Our analysis reveals that conventional aggregate metrics obscure regional heterogeneity, demonstrating that cognitive domain interventions yield significant specificity. Experimental results show that each expert model achieves superior fMRI prediction accuracy within its corresponding brain regions. This finding is consistently validated across three foundation models and multiple datasets, establishing a novel paradigm for fine-grained neural-linguistic alignment.

Abstract

Large language models (LLMs) can predict human brain activity across a variety of brain regions during natural language comprehension. Typically, however, LLM-brain alignment is measured using one model for different regions of the brain, and then model performance is summarized across regions. This one-model-fits-all approach ignores the functional specialization of brain regions. In this study, we assess whether a model oriented toward a particular cognitive domain aligns better with the brain system dedicated to that domain. Through prompting and fine-tuning, we first build expert LLM variants for six domains: sensory, spatial, numerical, reasoning, social, and abstract processing. We then examine whether each expert best predicts activity in the brain region associated with the corresponding cognitive domain. Consistent with our hypotheses, each expert's representations align more closely with the brain system most associated with the matching domain than do other experts. This holds under both prompting and fine-tuning, across three base models and three fMRI datasets. In a series of control analyses, we show that this model-brain alignment is specific to cognitive domain interventions; non-cognitive and surface-level interventions do not result in comparable alignment. Specializing models shifts regional alignment while leaving aggregate prediction accuracy largely unchanged, suggesting that summarizing alignment across regions may obscure regional differences in performance for specific models.

Problem

• Large Language Models
• Brain Alignment
• Cognitive Specialization
• fMRI
• Functional Specialization

Innovation

• Cognitive Expert Language Models
• Brain-Model Alignment
• Functional Specialization
• Fine-tuning and Prompting
• fMRI