🤖 AI Summary
This study investigates how input representations—specifically symbolic tokens, index-based “oracle” encodings, and entanglement-aware vectors—affect the binding capacity of miniature Transformers in compositional generalization. Models with 6–10K parameters are trained on fully enumerated finite factorized worlds, ensuring strict information-matching and eliminating sampling variance. The findings reveal that zero-shot binding performance falls below random chance across all input paths, with distinct failure modes: symbolic inputs lose answers during readout, index-based inputs suffer from incorrect binding, and entangled inputs inherit input readability limitations. Few-shot learning efficiency is jointly determined by parameter sharing and encoding readability inherent to each input path. These results challenge the assumption that clean oracle encodings are optimal, highlighting the critical role of input representation in compositional generalization.
📝 Abstract
How does the way information reaches a transformer -- as symbolic tokens, a clean per-factor "oracle" code, or an entangled perceptual vector -- shape whether it binds that information compositionally? We study ~6-10K-parameter transformers on finite factored worlds enumerated exhaustively, so every measurement covers the whole input space (zero sampling variance) and the informative routes are information-matched (exact Bayes ceiling 1.0). We report four findings. (1) Endpoint invariance: on held-out binding queries no informative route reaches converged zero-shot composition -- each ends at or below chance despite a ceiling of 1.0, so within a bounded sweep the failure reflects inductive bias under a lookup-sufficient objective, not missing information. (2) A two-factor account of few-shot binding: sample efficiency is best explained by input-pathway parameter sharing and code readability; a dimension-matched control and a graded readability sweep isolate readability from input dimension, and the clean oracle is not the most sample-efficient readable route. (3) A double dissociation: early in training, distributed -- but not index-like -- codes pass through a transient above-chance phase (tracking code format), while few-shot efficiency tracks pathway sharing. (4) Failure anatomy: symbolic routes lose the answer at the readout; index routes mis-bind (the answer stays decodable, yet an input intervention shows the output tracks the wrong slot); entangled routes inherit their input's readability. The central claim is the two-factor account; the endpoint and anatomy results are diagnostic constraints. All code, manifests, and per-seed logs are released for exact reproduction.