transformer architecture

This work investigates the universal mechanism by which Transformers implicitly learn temporal mappings—such as linear and periodic patterns—in autoregressive next-token prediction, with a focus on how self-attention encodes causal sequence structure. We propose *causal kernel descent*, a novel theoretical framework that, for the first time, constructsively proves that causal Transformers can implicitly implement an online Kaczmarz-type algorithm via self-attention to approximate a dynamic context function (f) in a Hilbert space. We rigorously construct a Transformer architecture capable of exactly learning (f), and validate its efficacy through controlled sequence experiments under linear, exponential, and softmax attention. This work establishes the first provably correct, constructively universal theoretical framework for understanding how large language models intrinsically perform in-context reasoning—offering a new paradigm for analyzing their implicit inference capabilities.

This work clarifies the functional boundaries and necessity of core Transformer components—tokenization, embedding/un-embedding, masking, positional encoding, and padding—addressing widespread conceptual ambiguity in their mechanistic roles. Targeting ML engineers, we propose an incremental, invertibility-based analytical framework: using binary (0/1) sequences as probes, we systematically introduce each component via a “zero-one construction” and empirically validate its irreplaceability in the encode-decode pipeline. Implemented lightweightly in PyTorch, our framework supports manual attention matrix construction, explicit positional embedding injection, and interpretable mask design. Experiments demonstrate significantly improved conceptual accuracy among learners. Notably, we provide the first empirical verification that, in the absence of self-attention, positional encoding combined with padding alone suffices for basic length-aware tasks.

The impact of pooling operations on representational capacity and task performance in Transformer models has long been overlooked. Method: We establish, for the first time, theoretical expressivity bounds for pooling methods and propose a unified analytical framework that characterizes how distinct pooling strategies—e.g., [CLS], mean, and attention-weighted pooling—affect input discriminability, contextual modeling capability, and optimization dynamics. Our analysis spans three modalities—NLP, computer vision, and time series—and encompasses multiple attention variants across diverse downstream tasks. Contribution/Results: Empirical evaluation reveals that pooling choice significantly influences accuracy, gradient sensitivity, and convergence stability. Crucially, we identify task-agnostic, high-performing pooling patterns that generalize consistently across modalities and tasks. This work provides both theoretical foundations and practical guidelines for task-aware pooling design in Transformer architectures.

This work addresses the lack of a rigorous mathematical characterization of Transformers’ ability to model contextual relationships. The authors propose a measure-theoretic framework in which text is represented as a probability measure over a semantic embedding space, and inter-word contextual dependencies are modeled as coupling measures. They introduce the Sinkhorn Transformer architecture to approximate arbitrary continuous coupling functions. By integrating optimal transport theory with function approximation theory, the study establishes the first universal approximation theorem for Transformers in modeling contextual relations, proving that the proposed architecture can uniformly approximate any continuous coupling function. This result provides a formal theoretical guarantee for the semantic coupling mechanism and expressive power of Transformers.

This work addresses the limitations of Transformer-based models in long-sequence multivariate time series forecasting, particularly their inadequate input representation quality and structural modeling capacity. To overcome these challenges, the authors propose a two-stage framework: first, a convolutional neural network (CNN) extracts local dynamic features from fixed-length temporal segments and generates compact patch-level token embeddings; subsequently, a Transformer encoder with attention mechanisms models the global dependencies among these segments. By decoupling local feature extraction from global dependency modeling, the approach enhances both scalability and representational power. Experimental results on synthetic multivariate time series datasets demonstrate that the proposed method significantly outperforms CNN baselines under long input sequences and achieves performance comparable to state-of-the-art patch-based Transformer models.

Existing positional encodings rely on predefined sequential orders, limiting their ability to model non-sequential yet causally related features. To address this, we propose Causal-Aware Positional Encoding (CAPE): first, a generalized structural equation model learns a causal directed acyclic graph (DAG); second, the DAG is embedded in hyperbolic space to preserve both causal strength and specificity; third, a rotation-based transformation seamlessly integrates the encoding into Transformer self-attention. We theoretically prove that CAPE satisfies causal distance metric properties and exhibits generalization decay—key desiderata for causal representation learning—while remaining robust to positional perturbations. Empirical evaluations on synthetic and real-world datasets demonstrate that CAPE significantly enhances Transformer’s representational capacity for non-sequential causal data. The implementation is publicly available.

It remains unclear whether Transformers can algorithmically infer transition probabilities of Markov chains from context via next-token prediction alone—rather than relying on memorized patterns. Method: We train standard autoregressive Transformers on sequences generated by randomly sampled Markov chains with diverse transition matrices, systematically varying model size and training data scale. Contribution/Results: We identify dual thresholds in model capacity and data volume beyond which Transformers transition from pattern memorization to genuine context-driven estimation of transition probabilities. Incorporating explicit state embeddings significantly improves zero-shot generalization to unseen chain structures. Crucially, sufficiently large Transformers achieve accurate modeling of previously unobserved Markov chains without fine-tuning—demonstrating, for the first time, systematic evidence that Transformers can implicitly learn probabilistic algorithms from context. This provides foundational insight into the emergent algorithmic reasoning capabilities of large language models.

This work investigates the equivalence between context updates and parameter updates in Transformers. We propose a low-rank (rank-1) weight patching mechanism that implicitly encodes contextual influence as dynamic, input-dependent corrections to MLP layer weights. We establish, for the first time, a formal controllability-theoretic framework—proving that such context-to-parameter mapping is exactly realizable in MLP blocks satisfying mild structural conditions. Our formulation rigorously accommodates modern architectural components, including RMSNorm, gating mechanisms, and pre-/post-norm configurations, thereby unifying the explanation of context-driven parameter adaptation across Gemma, MoE, and deep stacked architectures. Experiments confirm exact equivalence of context effects under this mechanism in both Gemma-style modules and multi-layer models, with broad applicability across mainstream LLM architectures. This work provides a novel theoretical and mechanistic paradigm for understanding context awareness in large language models.

This work addresses the inherent conflict between in-context learning (ICL) and in-weight learning (IWL) in Transformer models by proposing CoQE, a dual-representation architecture that decouples contextual information and input samples at the representation level. CoQE explicitly encodes context into a task representation space and samples into a distinct sample representation space, leveraging dual-space linear modeling grounded in duality theory and an enhanced Transformer structure. Experiments on few-shot classification and pseudo-arithmetic tasks demonstrate that CoQE significantly improves ICL performance while effectively preserving IWL capabilities, thereby validating the efficacy and generality of its co-optimization strategy for both learning paradigms.

Existing studies model induction heads using fixed-structure Markov chains, failing to capture the context-dependent dynamism inherent in natural language. Method: We propose the “Interleaved Delayed Markov Chain” framework, enabling Transformers to adaptively select optimal causal structures across varying contexts. We identify and construct a selective induction head circuit wherein self-attention and in-context learning jointly orchestrate dynamic structural switching; we theoretically prove its convergence to the maximum-likelihood solution. Contribution/Results: Experiments on a three-layer Transformer demonstrate that the model accurately infers context-optimal time delays and precisely replicates target tokens, simultaneously selecting causal structure during sequence prediction. This work overcomes the limitations of static causal modeling and establishes a novel paradigm for understanding the dynamic inductive mechanisms underlying Transformer architectures.