🤖 AI Summary
Large language models (LLMs) excel at code generation, yet the impact of their compressed variants—such as those produced via quantization or knowledge distillation—on token representations of programming languages remains poorly understood, hindering deployment quality. This work systematically investigates how LLM tokenizers encode programming languages and introduces a novel “cold-start probability” analysis method that operates without explicit prompting. By integrating lexical distribution analysis, keyword coverage, and multidimensional evaluation metrics, the study provides the first comprehensive characterization of the subtle effects of compression strategies—including quantization, knowledge distillation, model scaling, and task-specific fine-tuning—on code token representations. The findings offer both theoretical grounding and empirical guidance for deploying high-quality, efficient code generation models.
📝 Abstract
Large Language Models (LLMs) have demonstrated exceptional code generation capabilities, yet their token-level mechanisms remain underexplored, particularly in compressed models. Through systematic analysis of programming language token representations, we characterize how programming languages are encoded in LLM tokenizers by analyzing their vocabulary distribution and keyword coverage patterns. We introduce a novel cold-start probability analysis method that provides insights into model behavior without requiring explicit prompts. Additionally, we present a comprehensive evaluation of how different model optimization techniques - including quantization, distillation, model scaling, and task-specific fine-tuning - affect token-level representations and code generation quality. Our experiments, supported by comprehensive probability distribution analysis and evaluation metrics, reveal critical insights into token-level behavior and provide empirically-validated guidelines for maintaining code generation quality under various optimization constraints. These findings advance both theoretical understanding of LLM code generation and practical implementation of optimized models in production environments.