Is the Reversal Curse a Binding Problem? Uncovering Limitations of Transformers from a Basic Generalization Failure

2025-04-02

AI Summary

This paper identifies the “reversal curse” in large language models (LLMs)—their systematic failure to learn reversible factual associations—as a fundamental limitation arising from concept-binding defects in the Transformer architecture: inconsistent and entangled concept representations, constituting a concrete AI manifestation of the cognitive science “binding problem.” Method: To address this, the authors propose the first JEPA-based (Joint Embedding Predictive Architecture) model, which eliminates the need for data augmentation or non-causal masking; it incorporates a memory layer with disentangled concept representations to enable parametric forward-chaining reasoning. Results: Experiments demonstrate complete elimination of the reversal curse, with substantial performance gains over state-of-the-art LLMs on large-scale arithmetic reasoning benchmarks. The approach achieves significantly improved generalization and robustness, validating its efficacy in overcoming architectural binding limitations inherent to standard autoregressive Transformers.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Despite their impressive capabilities, LLMs exhibit a basic generalization failure known as the Reversal Curse, where they struggle to learn reversible factual associations. Understanding why this occurs could help identify weaknesses in current models and advance their generalization and robustness. In this paper, we conjecture that the Reversal Curse in LLMs is a manifestation of the long-standing binding problem in cognitive science, neuroscience and AI. Specifically, we identify two primary causes of the Reversal Curse stemming from transformers' limitations in conceptual binding: the inconsistency and entanglements of concept representations. We perform a series of experiments that support these conjectures. Our exploration leads to a model design based on JEPA (Joint-Embedding Predictive Architecture) that for the first time breaks the Reversal Curse without side-stepping it with specialized data augmentation or non-causal masking, and moreover, generalization could be further improved by incorporating special memory layers that support disentangled concept representations. We demonstrate that the skill of reversal unlocks a new kind of memory integration that enables models to solve large-scale arithmetic reasoning problems via parametric forward-chaining, outperforming frontier LLMs based on non-parametric memory and prolonged explicit reasoning.

Problem

• Understanding the Reversal Curse in LLMs as a binding problem
• Identifying transformer limitations in conceptual binding and representation
• Developing a JEPA-based model to overcome the Reversal Curse

Innovation

• JEPA architecture breaks Reversal Curse
• Memory layers improve concept disentanglement
• Parametric forward-chaining enhances arithmetic reasoning