Towards an automated workflow in materials science for combining multi-modal simulative and experimental information using data mining and large language models

2025-02-18

AI Summary

In materials science literature, multimodal information—including text, figures, equations, and metadata—is often inaccessible and non-machine-readable, hindering automated property extraction, contextual understanding, and efficient retrieval. Method: This paper introduces the first vision-language large model (VLLM) collaborative parsing framework tailored for materials science, enabling end-to-end structured extraction of multimodal content. It constructs a machine-readable knowledge base integrating local proprietary data and designs a retrieval-augmented generation (RAG)-enhanced large language model (LLM) question-answering system to support domain-specific reasoning, such as microstructure analysis. The methodology integrates vision Transformers, language Transformers, NLP, and data mining techniques. Contribution/Results: Experimental evaluation demonstrates significant improvements in retrieval speed and contextual recognition accuracy. The framework achieves fully automated material property extraction and high-accuracy QA responses in face-centered cubic (FCC) single-crystal microstructure analysis, establishing a scalable foundation for intelligent materials literature processing.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

To retrieve and compare scientific data of simulations and experiments in materials science, data needs to be easily accessible and machine readable to qualify and quantify various materials science phenomena. The recent progress in open science leverages the accessibility to data. However, a majority of information is encoded within scientific documents limiting the capability of finding suitable literature as well as material properties. This manuscript showcases an automated workflow, which unravels the encoded information from scientific literature to a machine readable data structure of texts, figures, tables, equations and meta-data, using natural language processing and language as well as vision transformer models to generate a machine-readable database. The machine-readable database can be enriched with local data, as e.g. unpublished or private material data, leading to knowledge synthesis. The study shows that such an automated workflow accelerates information retrieval, proximate context detection and material property extraction from multi-modal input data exemplarily shown for the research field of microstructural analyses of face-centered cubic single crystals. Ultimately, a Retrieval-Augmented Generation (RAG) based Large Language Model (LLM) enables a fast and efficient question answering chat bot.

Problem

• Automated workflow for materials science
• Machine-readable data from scientific literature
• Accelerated information retrieval using LLMs

Innovation

• Automated workflow for data extraction
• Natural language processing techniques
• Retrieval-Augmented Generation LLM