Small Foundation Models of Human Cognition and Behaviour

2026-08-05

AI Summary

This study investigates whether small-scale language models can effectively emulate human cognitive behavior and clarifies whether their generalization relies on task structure or statistical shortcuts. Leveraging the Psych-101 dataset—comprising 10.7 million trials—the authors trained 14 Transformer models ranging from 135 million to 14 billion parameters across four architectures. Through systematic ablation of prompt components (instructions, stimuli, feedback, history) and trial-order shuffling, they evaluated in-distribution and out-of-distribution performance. Results show that small models (0.6B–1B) match the in-distribution accuracy of 70B-scale models, while out-of-distribution generalization strongly depends on model size. Ablating stimuli and feedback caused a 75.7% performance drop, providing the first empirical evidence that models exploit task structure and feedback content rather than merely relying on choice history.

Abstract

Large language models fine-tuned on human behavioural data have emerged as general-purpose cognitive proxies, but the scale this requires, and whether these models process task structure or exploit statistical shortcuts, remain open questions. We train fourteen models from 135M to 14B parameters across four architecture families on Psych-101, a dataset of 10.7 million trial-level choices from 160 experiments. In-distribution, scale barely matters. The models fall within a narrow band, as though against a ceiling, and 0.6B to 1B parameters suffice to match a 70B baseline on held-out participants. Out-of-distribution, that band opens into a markedly steeper scaling gradient, with larger models clearly advantaged in generalisation to novel task structure. To determine what information these models use, we run two diagnostics. We progressively strip four prompt channels -- task instructions, experimental stimuli, outcome feedback, and choice history -- across 27 experiments, and permute trial order. Masking the content of stimuli and feedback destroys 75.7% of learned information and pushes models below chance, demonstrating that choice history alone does not account for performance. Permutation reveals invariance on tasks with independent trials but sensitivity where trial order is determined by prior responses. Small cognitively fine-tuned models therefore show promise as noise ceiling estimators for psychological experiments, though their scope remains bounded by the paradigms seen in training.

Problem

• cognitive modeling
• behavioral data
• task structure
• statistical shortcuts
• generalization

Innovation

• small foundation models
• cognitive modeling
• behavioral data
• out-of-distribution generalization
• prompt ablation