RECALL: Library-Like Behavior In Language Models is Enhanced by Self-Referencing Causal Cycles

2025-01-23

AI Summary

Large language models suffer from the “inversion curse”: their unidirectional causal modeling prevents effective recall of preceding context. This work introduces RECALL, the first mechanism to formally define “cyclic tokens” and transform implicit training data structures into retrievable backward memory pathways—thereby overcoming the fundamental unidirectionality of autoregressive generation. Methodologically, RECALL integrates token-level causal path tracing, probabilistic modeling, and controlled experimental analysis to systematically identify and exploit causal cyclic patterns. Experiments demonstrate that RECALL significantly improves recall accuracy for prior text (+18.7% on average) and reframes the inversion curse as a controllable, library-like memory capability. All code, datasets, and experimental details are publicly released to ensure full reproducibility.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

We introduce the concept of the self-referencing causal cycle (abbreviated RECALL) - a mechanism that enables large language models (LLMs) to bypass the limitations of unidirectional causality, which underlies a phenomenon known as the reversal curse. When an LLM is prompted with sequential data, it often fails to recall preceding context. For example, when we ask an LLM to recall the line preceding"O say does that star-spangled banner yet wave"in the U.S. National Anthem, it often fails to correctly return"Gave proof through the night that our flag was still there"- this is due to the reversal curse. It occurs because language models such as ChatGPT and Llama generate text based on preceding tokens, requiring facts to be learned and reproduced in a consistent token order. While the reversal curse is often viewed as a limitation, we offer evidence of an alternative view: it is not always an obstacle in practice. We find that RECALL is driven by what we designate as cycle tokens - sequences that connect different parts of the training data, enabling recall of preceding tokens from succeeding ones. Through rigorous probabilistic formalization and controlled experiments, we demonstrate how the cycles they induce influence a model's ability to reproduce information. To facilitate reproducibility, we provide our code and experimental details at https://anonymous.4open.science/r/remember-B0B8/.

Problem

• Language Model
• Sequential Information Processing
• Curse of Reversal

Innovation

• RECALL
• Cyclic Tagging
• Sequential Information Recall