tokenizer design

Existing tokenizer transfer methods for large language models (LLMs) struggle to adapt pretrained tokenizers to low-resource languages or scripts, as they typically rely on semantic heuristics for initialization while neglecting higher-level model dynamics—leading to suboptimal adaptation. This work proposes the Attention Influence Modeling (AIM) framework, which leverages inter-token attention communication patterns from the source model as supervisory signals to guide both initialization and optimization of target-language embeddings. AIM integrates attention behavior distillation with lightweight pre-warmup training prior to standard language modeling, enabling efficient tokenizer adaptation. Experiments show that AIM restores over 90% of the original model’s performance using only a few GPU hours, significantly outperforming mainstream baselines across diverse low-resource and cross-script languages. To our knowledge, this is the first tokenizer transfer approach driven explicitly by internal model dynamics.

Existing graph tokenization methods exhibit limitations in hierarchical structure modeling and task adaptability: quantization strategies are often fixed or task-agnostic, leading to imbalanced structural representation and hindering dynamic multi-scale contribution adjustment without retraining the encoder. This paper proposes HQ-Graph, a Hierarchical Quantization Graph tokenization framework that enables dynamic multi-scale graph structural aggregation under a frozen encoder via a lightweight self-weighted gating mechanism, supporting task-adaptive discrete representation learning. Its core innovation lies in the organic integration of hierarchical quantization, discrete representation, learnable gating, and multi-scale aggregation. Experiments demonstrate that HQ-Graph consistently outperforms strong baselines on node classification and link prediction tasks, achieving superior performance at comparable computational cost—thereby balancing expressive power and parameter efficiency.

To address the challenge of jointly achieving multi-scale semantic modeling and computational efficiency in Transformer-based large language models, this paper proposes the Hierarchical Lexical Manifold Projection (HLMP) mechanism, integrating structured hierarchical embeddings into the standard Transformer architecture. HLMP employs lexical-level manifold projection to unify local syntactic structure and global semantic relations within token representations, enabling smooth cross-scale semantic transitions; it further incorporates an attention enhancement module to improve contextual adaptability and robustness against perturbations. Evaluated on linguistic benchmarks—including GLUE and SuperGLUE—HLMP achieves average accuracy gains of 1.8–3.2% while reducing inference latency by 12%. Moreover, it significantly enhances representation consistency and structural interpretability on domain-specific texts and adversarial examples. Collectively, HLMP establishes a novel paradigm for efficient, interpretable, hierarchical semantic modeling.

This paper addresses the fundamental challenge that large language models (LLMs) cannot natively process graph-structured data. To bridge this gap, we propose LLM4graph—a unified framework introducing the first systematic taxonomy of Graph2Text and Graph2Token paradigms, and identifying four core challenges in graph-to-text/token conversion. Methodologically, LLM4graph integrates graph encoding, structured serialization, prompt engineering, and LLM adaptation techniques to achieve multi-granular, semantics-preserving graph representation transformation. Key contributions include: (1) a principled LLM selection guideline tailored to problem characteristics, hardware constraints, and task requirements; (2) a comprehensive survey of over 100 works in graph–LLM integration; and (3) distillation of five critical future research directions. This work establishes a theoretical foundation, technical methodology, and practical paradigm for the emerging intersection of graph learning and large language modeling.

It remains unclear whether current graph-tokenized large language models genuinely comprehend graph tokens embedded within natural language. This work proposes GTEval—the first systematic evaluation framework for graph token understanding—integrating unified modeling, instruction transformation strategies, attention analysis, and instruction tuning, with experiments conducted across six representative models. The study reveals that existing models are generally either overly sensitive or insensitive to instruction variations and heavily rely on textual reasoning, struggling to effectively leverage graph tokens. Although instruction tuning yields modest improvements, significant bottlenecks persist in graph token comprehension. This research establishes a new evaluation benchmark and analytical perspective for modeling the integration of graphs and language.

The precise mechanisms by which subword tokenization enhances language model performance remain unclear. This work systematically disentangles the independent effects of subword tokenization on training efficiency and linguistic priors within a controlled byte-level pretraining framework, by simulating subword boundaries, vocabulary scaling, and throughput variations. For the first time, subword boundaries are explicitly modeled as an inductive bias, revealing that their combination with increased training throughput constitutes the core advantage of subword-based models. These findings provide both theoretical grounding and practical guidance for hybrid pretraining approaches that integrate byte-level and subword representations.

This work proposes a sequential fusion approach based on large language models (LLMs) to address the challenges of integrating cross-lingual knowledge graphs, which suffer from semantic heterogeneity and structural complexity. By leveraging LLMs as a universal semantic bridge for the first time, the method linearizes knowledge graph triples into natural language sequences and integrates cross-lingual entity alignment with relation mapping to enable modular, scalable, and continuous fusion of multi-source heterogeneous graphs. Experimental results on the DBP15K dataset demonstrate that the proposed approach effectively supports sequential aggregation of multilingual knowledge graphs while achieving strong performance and scalability.

This work addresses the challenge of efficiently converting graph-structured data into sequences compatible with general-purpose Transformer models. The authors propose a novel graph tokenization framework that, for the first time, integrates reversible graph serialization—guided by global substructure frequency statistics—with Byte Pair Encoding (BPE) to produce compact token representations that preserve structural semantics while remaining amenable to sequence-based architectures. Notably, this approach requires no modifications to standard Transformer backbones such as BERT, enabling direct application to graph data. Evaluated across 14 established graph benchmark datasets, the method significantly outperforms both conventional graph neural networks and specialized graph Transformers, achieving state-of-the-art performance.

This work addresses the challenge of efficiently combining the capabilities of multiple fine-tuned large language models without resorting to costly retraining or computationally expensive ensembles. The authors propose FUSE, a four-dimensional taxonomy that systematically unifies the theoretical foundations of model merging—such as loss landscape geometry and mode connectivity—with practical strategies including weight averaging, task vector arithmetic, sparsity-enhanced methods, mixture-of-experts architectures, and evolutionary optimization. By establishing a comprehensive, training-free framework for model merging, this study organizes existing open-source tools and evaluation benchmarks, while also identifying critical theoretical gaps, scalability challenges, and the need for standardization. The resulting knowledge体系 offers a structured roadmap to guide future research in efficient and effective model fusion.