Representational Homomorphism Predicts and Improves Compositional Generalization In Transformer Language Model

2026-01-26

AI Summary

Neural networks still struggle with compositional generalization—the ability to interpret novel combinations of familiar elements—and lack a deep understanding of the representational mechanisms underlying this failure. This work introduces, for the first time, the concept of algebraic homomorphism into language model analysis and proposes Homomorphism Error (HE), a quantifiable metric that measures the deviation of Transformer hidden states from an ideal algebraic structure. By defining modifier- and sequence-level HE on SCAN-like tasks and combining it with noise injection, data ablation, and HE-regularized training, the study demonstrates a strong correlation between HE and out-of-distribution accuracy (R² = 0.73). HE regularization significantly reduces error (p < 0.01) and improves compositional generalization (p = 0.023), establishing HE as both a diagnostic tool and a trainable objective, thereby offering a new pathway toward enhancing systematic generalization in neural models.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Compositional generalization-the ability to interpret novel combinations of familiar components-remains a persistent challenge for neural networks. Behavioral evaluations reveal when models fail but offer limited insight into why failures arise at the representational level. We introduce Homomorphism Error (HE), a structural metric that quantifies deviations from approximate homomorphisms between the expression algebra and a model's hidden-state space. We instantiate HE for two compositional operators in SCAN-style tasks: modifier HE for unary composition and sequence HE for binary composition, measured by learning representation-level operators that predict composed representations from their parts. Across controlled experiments with small decoder-only Transformers, HE predicts out-of-distribution (OOD) compositional generalization under noise injection, achieving R^2 = 0.73 correlation between modifier HE and OOD accuracy. Ablations show that model depth has minimal effect on either HE or OOD accuracy, training data coverage exhibits threshold effects (insufficient coverage sharply increases HE and degrades OOD performance), and randomly inserted noise tokens systematically increase HE. Finally, we test if HE-regularized training improves OOD accuracy. Experiment shows that explicitly enforcing low modifier HE during training significantly reduces modifier HE (p = 1.1x10-4) and sequence HE (p = 0.001) and yields a statistically significant improvement in OOD accuracy (p = 0.023). Together, these results indicate the potential of HE to be both a diagnostic and an actionable training signal for improving compositional generalization. Code to reproduce our experiments is open-sourced.

Problem

• compositional generalization
• representational homomorphism
• out-of-distribution generalization
• neural network representations
• SCAN tasks

Innovation

• Homomorphism Error
• Compositional Generalization
• Transformer Language Models
• Structural Regularization
• Out-of-Distribution Generalization