Mic-hackathon 2024: Hackathon on Machine Learning for Electron and Scanning Probe Microscopy

2025-06-10

AI Summary

Microscopy data analysis is hindered by fragmented software ecosystems, lack of standardized benchmarks, and a disconnect between machine learning (ML) and microscopy communities—limiting its impact in materials science and physical discovery. To address this, we propose an interdisciplinary collaborative paradigm and introduce the first microscope digital twin framework. This framework integrates a standardized multimodal benchmark dataset (spanning electron and probe microscopy), FAIR-compliant metadata specifications, and a real-time API-driven ML agent control interface. Built upon the Python ecosystem, vendor APIs (Thermo Fisher, Bruker), PyTorch/TensorFlow, and OMERO standards, it delivers an open-source ML benchmark suite. Our approach reduces image analysis turnaround time significantly, enables three automated microscopy operation prototypes, and supports five cross-institutional collaborative workflows—advancing microscopy intelligence from offline modeling toward closed-loop autonomous experimentation.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Web Mining and Content Analysis: Machine learning and data science for the Web

Abstract

Microscopy is a primary source of information on materials structure and functionality at nanometer and atomic scales. The data generated is often well-structured, enriched with metadata and sample histories, though not always consistent in detail or format. The adoption of Data Management Plans (DMPs) by major funding agencies promotes preservation and access. However, deriving insights remains difficult due to the lack of standardized code ecosystems, benchmarks, and integration strategies. As a result, data usage is inefficient and analysis time is extensive. In addition to post-acquisition analysis, new APIs from major microscope manufacturers enable real-time, ML-based analytics for automated decision-making and ML-agent-controlled microscope operation. Yet, a gap remains between the ML and microscopy communities, limiting the impact of these methods on physics, materials discovery, and optimization. Hackathons help bridge this divide by fostering collaboration between ML researchers and microscopy experts. They encourage the development of novel solutions that apply ML to microscopy, while preparing a future workforce for instrumentation, materials science, and applied ML. This hackathon produced benchmark datasets and digital twins of microscopes to support community growth and standardized workflows. All related code is available at GitHub: https://github.com/KalininGroup/Mic-hackathon-2024-codes-publication/tree/1.0.0.1

Problem

• Lack of standardized code ecosystems for microscopy data analysis
• Gap between ML and microscopy communities limits impact
• Inefficient data usage and extensive analysis time

Innovation

• Developed standardized code ecosystems for microscopy data
• Created real-time ML APIs for microscope automation
• Produced benchmark datasets and digital twins