Attention Function as an Intrinsic Inductive Bias: How Models' Behavior Diverges in Novel Contexts

2026-09-30

AI Summary

This study investigates whether Transformer attention activation functions constitute intrinsic inductive biases that dominate model behavior in out-of-distribution (OOD) scenarios. To this end, we propose Mixture of Function Attention (MoFA), a parameter-free architecture that fixes the ratio of Softmax to Sigmoid attention heads based on GPT-2, and conduct multi-ratio and zero-shot distribution shift experiments. Our findings reveal that activation functions act as intrinsic priors: they are suppressed within the training domain yet re-emerge and govern behavior under weakly constrained OOD conditions. Empirically, while varying head ratios yield no significant performance differences in-domain, perplexity gaps expand by an order of magnitude across 15 OOD settings, with the optimal ratio accounting for 78.3% of the performance variance.

Abstract

Developmental psychology holds that certain priors are given to infants prior to experience rather than induced from data, and that the influence of such priors is suppressed under strong, well-constrained conditions but reasserts itself under weak ones. We ask whether an analogous principle holds for the Transformer: can the activation function given to attention heads serve as an intrinsic inductive bias? We propose Mixture of Function Attention (MoFA), a parameter-free modification to multi-head attention that fixes a ratio of softmax and sigmoid heads before training. Across five ratios, a 124M-parameter GPT-2 model, and five seeds, we find that this given ratio has little effect in-distribution -- differences between ratios are statistically negligible for moderate mixtures and remain small even at the extremes -- but its influence re-emerges sharply under zero-shot distribution shift across 15 out-of-distribution domains. Perplexity gaps between ratios widen by more than an order of magnitude on several domains, and the best-performing ratio tracks a single axis of domain structure, separating short, informal text (softmax-favoring) from technical, long-form text (sigmoid-favoring), that explains 78.3% of the variance in domain response. This reorganization is visible at the head level: sigmoid heads show an accelerating drop in attention entropy as their ratio increases, while softmax heads respond more modestly, yielding a consistent division of labor between the two head types. Our results suggest that activation choice functions as a given prior whose influence is masked in-distribution and re-emerges out-of-distribution.

Problem

• inductive bias
• attention activation function
• distribution shift
• Transformer
• out-of-distribution

Innovation

• Mixture of Function Attention
• Inductive Bias
• Out-of-Distribution Generalization
• Attention Mechanism
• Zero-Shot Transfer