The Life of a Token: from Words to Bits on the Wire

2026-09-17

AI Summary

本文通过分析模型架构、分词和并行策略，解释了大型语言模型如何将文本转化为网络通信流，并探讨了训练过程中的通信需求。

Abstract

Large Language Models (LLMs) transform vast collections of unstructured text into semantic patterns used for language generation and reasoning tasks. Behind their ease of use lies a complex process: words become tokens, tokens become vectors, and vectors ultimately give rise to streams of bits that flow through High-Performance Computing (HPC) systems. As modern LLMs grow to billions or trillions of parameters, this path increasingly unfolds across thousands of interconnected accelerators, making the underlying communication fabric a critical and often opaque component of model training. This tutorial aims to walk the reader through the journey from words to network traffic, shedding light on how language is translated into communication flows within HPC training systems. Using concrete examples from Dante's Divine Comedy, we illustrate how model architecture, tokenization, embeddings, and parallelization strategies shape the volume, structure, and timing of data exchanged across the network. We combine architectural analysis with analytical traffic models and numerical examples to characterize the communication requirements of LLM training. We try to demystify how words travel across the network and provide practical insights into the network requirements needed to support the journey from text to trained model.

Problem

• Large Language Models
• communication fabric
• HPC systems
• tokenization
• network traffic

Innovation

• Large Language Models
• High-Performance Computing
• Communication Fabric
• Tokenization
• Parallelization Strategies