Incorporating Domain Knowledge into Materials Tokenization

2025-06-09

AI Summary

Generic tokenization methods cause excessive fragmentation, semantic distortion, and loss of structural integrity for material entities in materials science text. To address this, we propose MATTER, a domain-adaptive tokenization framework. Its core contributions are: (1) MatDetector—a material-knowledge-base-pretrained model that accurately identifies material entity boundaries; and (2) a knowledge-driven token reordering and merging mechanism that explicitly preserves semantic fidelity and structural consistency of material concepts. MATTER integrates domain-specific knowledge injection with customized tokenization strategies. Evaluated on generation and classification tasks, MATTER achieves average performance gains of 4.0% and 2.1%, respectively, significantly outperforming mainstream tokenization approaches. It establishes an interpretable, high-fidelity foundational tokenization paradigm for materials NLP.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Matrix & Tensor Methods
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies

Abstract

While language models are increasingly utilized in materials science, typical models rely on frequency-centric tokenization methods originally developed for natural language processing. However, these methods frequently produce excessive fragmentation and semantic loss, failing to maintain the structural and semantic integrity of material concepts. To address this issue, we propose MATTER, a novel tokenization approach that integrates material knowledge into tokenization. Based on MatDetector trained on our materials knowledge base and a re-ranking method prioritizing material concepts in token merging, MATTER maintains the structural integrity of identified material concepts and prevents fragmentation during tokenization, ensuring their semantic meaning remains intact. The experimental results demonstrate that MATTER outperforms existing tokenization methods, achieving an average performance gain of $4%$ and $2%$ in the generation and classification tasks, respectively. These results underscore the importance of domain knowledge for tokenization strategies in scientific text processing. Our code is available at https://github.com/yerimoh/MATTER

Problem

• Improves tokenization for materials science language models
• Reduces fragmentation and semantic loss in material concepts
• Integrates domain knowledge to maintain structural integrity

Innovation

• Integrates material knowledge into tokenization
• Uses MatDetector trained on knowledge base
• Re-ranks tokens to prioritize material concepts