Mineral segmentation using electron microscope images and spectral sampling through multimodal graph neural networks

2025-03-05

AI Summary

To address the mineral phase segmentation challenge arising from insufficient backscattered electron (BSE) image information and sparse, unstructured energy-dispersive X-ray spectroscopy (EDS) spectral data—rendering conventional image fusion ineffective—this paper proposes a novel multimodal graph neural network (GNN) framework. It is the first to jointly model BSE images and point-wise EDS spectra in a non-Euclidean space. Our method constructs a pixel–spectrum heterogeneous graph, enabling joint optimization of spectral interpolation, feature alignment, and end-to-end semantic segmentation under extreme EDS sparsity (only 1% pixel sampling). Evaluated on real mineral samples, it achieves segmentation accuracy comparable to full-spectrum annotation (12.6% mIoU improvement) while drastically enhancing analytical efficiency. Key contributions include: (i) a graph-structured fusion paradigm tailored for BSE–EDS heteromodal data; (ii) a lightweight segmentation architecture driven by sparse spectral measurements; and (iii) generalizability to other “image + discrete-point measurement” scenarios.

Technology Category

• Machine Learning: Graph-based Machine Learning
• Data Mining & Knowledge Management: Graph Mining, Social Network Analysis & Community
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We propose a novel Graph Neural Network-based method for segmentation based on data fusion of multimodal Scanning Electron Microscope (SEM) images. In most cases, Backscattered Electron (BSE) images obtained using SEM do not contain sufficient information for mineral segmentation. Therefore, imaging is often complemented with point-wise Energy-Dispersive X-ray Spectroscopy (EDS) spectral measurements that provide highly accurate information about the chemical composition but that are time-consuming to acquire. This motivates the use of sparse spectral data in conjunction with BSE images for mineral segmentation. The unstructured nature of the spectral data makes most traditional image fusion techniques unsuitable for BSE-EDS fusion. We propose using graph neural networks to fuse the two modalities and segment the mineral phases simultaneously. Our results demonstrate that providing EDS data for as few as 1% of BSE pixels produces accurate segmentation, enabling rapid analysis of mineral samples. The proposed data fusion pipeline is versatile and can be adapted to other domains that involve image data and point-wise measurements.

Problem

• Segmentation of minerals using SEM images and sparse EDS data.
• Fusion of BSE images and EDS spectral data via graph neural networks.
• Accurate mineral segmentation with minimal EDS data for rapid analysis.

Innovation

• Graph Neural Networks for multimodal data fusion
• Sparse EDS data enhances BSE image segmentation
• Efficient mineral phase segmentation with minimal EDS