transformer encoding

This work clarifies the functional boundaries and necessity of core Transformer components—tokenization, embedding/un-embedding, masking, positional encoding, and padding—addressing widespread conceptual ambiguity in their mechanistic roles. Targeting ML engineers, we propose an incremental, invertibility-based analytical framework: using binary (0/1) sequences as probes, we systematically introduce each component via a “zero-one construction” and empirically validate its irreplaceability in the encode-decode pipeline. Implemented lightweightly in PyTorch, our framework supports manual attention matrix construction, explicit positional embedding injection, and interpretable mask design. Experiments demonstrate significantly improved conceptual accuracy among learners. Notably, we provide the first empirical verification that, in the absence of self-attention, positional encoding combined with padding alone suffices for basic length-aware tasks.

This paper investigates whether a single-layer Transformer without positional encoding possesses the universal approximation property (UAP) for vocabulary-in-context learning (VICL). Theoretically, we prove that in the absence of positional encoding, the model cannot achieve VICL-UAP; however, introducing positional encodings satisfying specific spectral conditions—such as sinusoidal encoding—strictly restores UAP. Our analysis is grounded in function approximation theory, where we formally model and analyze VICL capability via mathematical characterization of representational capacity. This work establishes, for the first time, a necessary and sufficient framework linking the existence of positional encoding to VICL-UAP. The results demonstrate, from an approximation-theoretic perspective, that positional encoding is both *necessary and sufficient* for VICL-UAP—not merely a heuristic aid for sequence modeling, but a fundamental theoretical prerequisite for contextual generalization. This provides a novel paradigm for understanding the essential role of positional information in Transformers.

Scaling Transformer models is prohibitively expensive due to fixed-parameter linear projection layers; architectural modifications necessitate full retraining. Method: We propose TokenFormer, the first architecture introducing *parameter tokenization*, which models model parameters as learnable tokens and replaces all linear layers with token-parameter self-attention—unifying parameter and input token representations in a shared latent space. Contribution/Results: Our method enables zero-shot, progressive parameter expansion without retraining, overcoming classical scaling bottlenecks. Without altering network topology, we scale model parameters from 124M to 1.4B while matching the performance of fully trained baselines, achieving substantial training cost reduction. The code and models are publicly released.

This work investigates how Transformers dynamically acquire inductive capabilities during in-context learning (ICL), specifically focusing on the role of “inductive heads” in transitioning from local n-gram pattern recognition to modeling long-range dependencies. Method: We combine theoretical approximation analysis, synthetic task training dynamics modeling, attention decomposition, and mixed-objective trajectory tracking across training. Contribution/Results: We formally characterize the generalized inductive head mechanism for the first time, revealing a sharp, non-gradual phase transition—from 4-gram modeling to inductive head emergence—during training. We quantify the layer- and head-specific contributions to long-range dependency capture and demonstrate that inductive heads constitute the core architectural substrate underlying ICL emergence. Our study provides the first full-training-dynamics evidence and an interpretable framework for understanding how large language models dynamically generalize, bridging mechanistic analysis with empirical learning trajectories.

The evolution of embedding techniques from word vectors to multimodal representations remains fragmented, lacking a unified framework that integrates advances across linguistic, cross-lingual, personalized, and multimodal domains—particularly for embodied multimodal learning in large language models. Method: We systematically survey static and contextual language representations, cross-lingual and personalized modeling, sentence/document embeddings, and multimodal fusion in vision, robotics, and cognitive science. We synthesize recent progress in interpretability, model compression, numerical encoding, and bias mitigation, and propose a novel paradigm emphasizing strong alignment across non-textual modalities and scalable training. Contributions: We construct a comprehensive knowledge graph of end-to-end embedding technologies—from Word2Vec and BERT to GPT, generative topic models, and multimodal alignment/distillation methods—identifying key technical bottlenecks and ethical challenges. This work delivers the first systematic roadmap for multimodal, embodied learning in foundation models.

This work investigates how different sequence representations—such as bytes, characters, and subwords—affect the information acquisition capacity of Transformer models under a fixed context window, a question that remains poorly understood. From an information-theoretic perspective, the paper introduces the notion of “fragmentation” and formally demonstrates that it inherently increases the log-loss of the optimal finite-context model. It establishes theoretical guarantees linking tokenization compression rates to the reliability of source context coverage, thereby constructing the first information-theoretic framework for representation selection in finite-context settings. Through Markov source modeling and comparative analysis of various tokenization strategies—including BPE, WordPiece, and byte-level methods—the study reveals the theoretical underpinnings of performance differences observed in models like ByT5 and CANINE, and proposes practical metrics to evaluate the effective context coverage of real-world tokenizers.

This work addresses the high computational cost in large language model (LLM) inference caused by contextual redundancy. We propose ARC-Encoder, a general-purpose, architecture- and parameter-agnostic context compression method that requires no modification to the target LLM. ARC-Encoder learns continuous, compact textual representations to replace original token embeddings, directly injecting them into the decoder’s input layer. Its core contribution is a unified encoder architecture—designed for broad compatibility across diverse LLM families—combined with continuous representation learning and a systematic training strategy, enabling 4×–8× context compression without fine-tuning the target model. Experiments demonstrate significant reductions in inference latency and GPU memory consumption across both instruction-tuned and base LLMs, with seamless plug-and-play deployment. ARC-Encoder achieves state-of-the-art performance in efficiency and practicality.

This work investigates the equivalence between context updates and parameter updates in Transformers. We propose a low-rank (rank-1) weight patching mechanism that implicitly encodes contextual influence as dynamic, input-dependent corrections to MLP layer weights. We establish, for the first time, a formal controllability-theoretic framework—proving that such context-to-parameter mapping is exactly realizable in MLP blocks satisfying mild structural conditions. Our formulation rigorously accommodates modern architectural components, including RMSNorm, gating mechanisms, and pre-/post-norm configurations, thereby unifying the explanation of context-driven parameter adaptation across Gemma, MoE, and deep stacked architectures. Experiments confirm exact equivalence of context effects under this mechanism in both Gemma-style modules and multi-layer models, with broad applicability across mainstream LLM architectures. This work provides a novel theoretical and mechanistic paradigm for understanding context awareness in large language models.

This work addresses the inherent conflict between in-context learning (ICL) and in-weight learning (IWL) in Transformer models by proposing CoQE, a dual-representation architecture that decouples contextual information and input samples at the representation level. CoQE explicitly encodes context into a task representation space and samples into a distinct sample representation space, leveraging dual-space linear modeling grounded in duality theory and an enhanced Transformer structure. Experiments on few-shot classification and pseudo-arithmetic tasks demonstrate that CoQE significantly improves ICL performance while effectively preserving IWL capabilities, thereby validating the efficacy and generality of its co-optimization strategy for both learning paradigms.

This work investigates the capability of Transformers to perform in-context arithmetic reasoning over algebraic sequences with dynamically shifting symbolic semantics—e.g., where variable mappings change in real time across distinct group structures. To this end, we propose a controllable group-distribution data generation strategy and a causal intervention testing framework, augmented by attention mechanism analysis and attribution methods. Our analysis reveals that Transformers spontaneously acquire three interpretable symbolic reasoning mechanisms: commutativity-preserving copying, identity element recognition, and closure-driven elimination—bypassing conventional geometric embedding paradigms. Empirically, the model achieves near-perfect (≈100%) accuracy on dynamic symbolic tasks and generalizes robustly to unseen algebraic groups. This constitutes the first empirical demonstration that Transformers possess intrinsic capacity for abstract symbolic manipulation and algebraic structure induction—without reliance on predefined semantic priors or architectural constraints.