MatterChat: A Multi-Modal LLM for Material Science

2025-02-18

AI Summary

This study addresses the challenge of fully integrating atomic-scale structural data with multimodal large language models (MLLMs) at native resolution to enhance inorganic material property prediction and scientific reasoning. To this end, we propose the first structure-aware bridging module that enables efficient, physics-consistent alignment between machine-learned atomic potentials (e.g., M3GNet) and Transformer-based language models. We further introduce a lightweight cross-modal joint fine-tuning paradigm that unifies atomic coordinates, crystal graphs, and textual inputs. Experiments demonstrate that our model significantly outperforms general-purpose LLMs—including GPT-4—across diverse tasks: material property prediction, synthesis pathway planning, and interpretable scientific question answering. Validated in real-world applications within energy and electronic materials domains, the framework establishes a new paradigm for physics-informed multimodal foundation models in materials science.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Understanding and predicting the properties of inorganic materials is crucial for accelerating advancements in materials science and driving applications in energy, electronics, and beyond. Integrating material structure data with language-based information through multi-modal large language models (LLMs) offers great potential to support these efforts by enhancing human-AI interaction. However, a key challenge lies in integrating atomic structures at full resolution into LLMs. In this work, we introduce MatterChat, a versatile structure-aware multi-modal LLM that unifies material structural data and textual inputs into a single cohesive model. MatterChat employs a bridging module to effectively align a pretrained machine learning interatomic potential with a pretrained LLM, reducing training costs and enhancing flexibility. Our results demonstrate that MatterChat significantly improves performance in material property prediction and human-AI interaction, surpassing general-purpose LLMs such as GPT-4. We also demonstrate its usefulness in applications such as more advanced scientific reasoning and step-by-step material synthesis.

Problem

• Integrating material structure with language models
• Enhancing human-AI interaction in material science
• Improving material property prediction accuracy

Innovation

• Multi-modal LLM integration
• Structure-aware material data
• Bridging module alignment