transformers

This work clarifies the functional boundaries and necessity of core Transformer components—tokenization, embedding/un-embedding, masking, positional encoding, and padding—addressing widespread conceptual ambiguity in their mechanistic roles. Targeting ML engineers, we propose an incremental, invertibility-based analytical framework: using binary (0/1) sequences as probes, we systematically introduce each component via a “zero-one construction” and empirically validate its irreplaceability in the encode-decode pipeline. Implemented lightweightly in PyTorch, our framework supports manual attention matrix construction, explicit positional embedding injection, and interpretable mask design. Experiments demonstrate significantly improved conceptual accuracy among learners. Notably, we provide the first empirical verification that, in the absence of self-attention, positional encoding combined with padding alone suffices for basic length-aware tasks.

Spanish-speaking researchers face high barriers to understanding Transformer models due to limited localized pedagogical resources and a scarcity of rigorous, accessible academic materials in Spanish. Method: This work constructs the first three-tiered knowledge framework—spanning mathematical foundations, algorithmic implementation, and multimodal applications—systematically explaining core components (self-attention, positional encoding, multi-head attention, layer normalization, and feed-forward networks), and extending them to cross-modal tasks including natural language processing, computer vision, speech processing, and reinforcement learning. It introduces a cognition-aware Spanish-language pedagogical framework integrating theoretical derivations, reproducible code, and domain-specific case studies. Contribution: This is the first structured, pedagogically grounded, and engineering-oriented dissemination of state-of-the-art Transformer theory in the Spanish-speaking academic community. It substantially lowers the conceptual and practical barriers to comprehension and reproduction, thereby advancing the deep localization and equitable adoption of foundational AI models beyond English-dominant research ecosystems.

Neural architecture search (NAS) remains hindered by heavy reliance on human expertise, predefined search spaces, and computationally expensive training-based evaluation. Method: This paper introduces the first end-to-end, Transformer-based neural architecture performance prediction framework. Its core innovation lies in fully tokenizing entire network architectures—integrating edge-free graph encoding with sequential representation—to enable zero-shot, training-free performance regression. Contribution/Results: The approach eliminates conventional NAS constraints on search space design and supports generative architecture exploration. Evaluated on DeepNets-1M, it achieves state-of-the-art performance prediction accuracy, reducing mean absolute error by 18.7% over the best proxy models and graph neural network baselines. This work establishes a new paradigm for fully automated, highly innovative neural network design.

To address the scalability bottlenecks of large language models arising from ever-increasing parameter and data requirements, this paper proposes an efficient decoder-only Transformer architecture. Methodologically, it employs a byte-level BPE tokenizer with a 128K vocabulary, integrated with Rotary Position Embeddings (RoPE), Grouped-Query Attention (GQA), RMSNorm, and SwiGLU activation—collectively reducing architectural redundancy and data dependency. Our key contribution is a lightweight yet highly effective model: trained on only 100 billion tokens with just 650 million parameters, it achieves 90% of the performance of a 1-billion-parameter baseline—reducing parameter count by 53% and training token volume by an order of magnitude. This demonstrates that careful architectural optimization enables a compelling trade-off between computational efficiency and modeling capability, without sacrificing downstream effectiveness.

Scaling Transformer models is prohibitively expensive due to fixed-parameter linear projection layers; architectural modifications necessitate full retraining. Method: We propose TokenFormer, the first architecture introducing *parameter tokenization*, which models model parameters as learnable tokens and replaces all linear layers with token-parameter self-attention—unifying parameter and input token representations in a shared latent space. Contribution/Results: Our method enables zero-shot, progressive parameter expansion without retraining, overcoming classical scaling bottlenecks. Without altering network topology, we scale model parameters from 124M to 1.4B while matching the performance of fully trained baselines, achieving substantial training cost reduction. The code and models are publicly released.

Existing efforts to encode algorithms directly into Transformer parameters are fragmented and lack systematicity, hindering both learning and reuse. Method: This paper introduces, for the first time, a unified “algorithm encoding recipe” framework that systematically integrates arithmetic implementations in feed-forward layers with dynamic data routing via self-attention, yielding a modular and composable Transformer construction methodology. Contribution: It bridges a critical gap between interpretable modeling and controllable architectural design of Transformers. The framework enables rigorous computational complexity analysis, formally verifiable architecture specification, and mechanistic interpretation of internal operations. By significantly lowering the barrier to algorithmic encoding—providing accessible onboarding for novices and structured, reusable building blocks for experts—it advances research and deployment of controllable, interpretable Transformers.

This study addresses the high computational cost and latency of Transformers in streaming automatic speech recognition (ASR), primarily caused by the self-attention mechanism. For the first time, we systematically evaluate and demonstrate that the self-attention module can be entirely removed from streaming ASR architectures without causing a significant increase in word error rate (WER). To this end, we propose two lightweight alternatives: replacing self-attention with deformable convolutions or simply omitting the module altogether. Experimental results show that both approaches substantially reduce computational complexity and inference latency while maintaining competitive recognition performance. These findings challenge the prevailing assumption that the Transformer architecture—and specifically its self-attention component—is indispensable for high-quality streaming ASR.

This work addresses the challenge of understanding backpropagation in Transformer architectures (e.g., GPT) for beginners. We systematically derive analytical gradients for core components—including token embeddings, multi-head self-attention, LayerNorm, and LoRA—using an index-free vectorized differentiation approach that avoids cumbersome subscript manipulations and yields lightweight, closed-form gradient expressions. To our knowledge, this is the first unified, end-to-end analytical derivation of full-chain backpropagation in Transformers, with explicit gradient formulas for LoRA fine-tuning. We accompany the analysis with a minimal runnable GPT implementation and provide closed-form solutions for all parameter updates. The results significantly enhance theoretical understanding of Transformer training dynamics and improve debugging capabilities, thereby establishing a rigorous foundation for pedagogy and interpretability research.

This work investigates whether the standard Transformer architecture is universally optimal across all tasks and proposes an architectural refinement that introduces task-specific inductive biases through learnable nonlinear components, such as GeLU or softmax. The approach preserves the original Transformer structure while replacing key activation functions with task-optimized counterparts learned during training. Experimental results demonstrate that this modification substantially improves learning speed, in- and out-of-distribution generalization, and training stability on algorithmic reasoning tasks. Consistent, albeit more modest, performance gains are also observed in language and code modeling, accompanied by enhanced cross-domain transfer capabilities. These findings indicate that the standard Transformer is not locally optimal for specific tasks and that incorporating task-tailored design elements—despite a trade-off in generality—can effectively boost performance.

This work addresses the limited parallelizability of conventional Transformers as depth increases, which hinders training efficiency for large-scale models. The authors introduce, for the first time, a multilayer parallel-in-time algorithm into Transformer training by modeling the network as a neural ordinary differential equation (neural ODE), enabling cross-layer parallelism in both forward and backward passes. To ensure stable convergence while maximizing computational efficiency, they further propose an error-monitoring mechanism with adaptive switching between serial and parallel execution modes. Experiments on BERT, GPT-2, Vision Transformers (ViT), and machine translation architectures demonstrate that the method significantly enhances parallel scalability and training speed for deep models without compromising pretraining or fine-tuning accuracy.