Foundation Models for Zero-Shot Segmentation of Scientific Images without AI-Ready Data

2025-06-30

AI Summary

To address the scarcity of annotated scientific images and poor adaptability of existing segmentation models, this paper introduces Zenesis—a training-free, code-free interactive zero-shot image segmentation platform. Methodologically, it innovatively integrates lightweight multimodal feature adaptation, natural language/point-box prompting, human-in-the-loop feedback optimization, and heuristic temporal enhancement, ensuring compatibility with scientific imaging modalities such as FIB-SEM. Evaluated on amorphous and crystalline catalyst images, Zenesis achieves accuracy of 0.947 and 0.987, IoU of 0.858 and 0.857, and Dice score of 0.923—substantially outperforming baselines including Otsu and SAM. Its core contribution is the first zero-shot segmentation framework tailored for scientific imagery, eliminating reliance on AI-ready annotations and enabling rapid, accurate, low-barrier analysis of high-value research images.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Zero-shot and prompt-based technologies capitalized on using frequently occurring images to transform visual reasoning tasks, which explains why such technologies struggle with valuable yet scarce scientific image sets. In this work, we propose Zenesis, a comprehensive no-code interactive platform designed to minimize barriers posed by data readiness for scientific images. We develop lightweight multi-modal adaptation techniques that enable zero-shot operation on raw scientific data, along with human-in-the-loop refinement and heuristic-based temporal enhancement options. We demonstrate the performance of our approach through comprehensive comparison and validation on challenging Focused Ion Beam Scanning Electron Microscopy (FIB-SEM) data of catalyst-loaded membranes. Zenesis significantly outperforms baseline methods, achieving an average accuracy of 0.947, an Intersection over Union (IOU) of 0.858, and a Dice score of 0.923 for amorphous catalyst samples and accuracy of 0.987, an IOU of 0.857, and a Dice score of 0.923 for crystalline samples. These results mark a substantial improvement over traditional methods like Otsu thresholding and even advanced models like Segment Anything Model (SAM) when used in isolation. Our results demonstrate that Zenesis is a powerful tool for scientific applications, particularly in fields where high-quality annotated datasets are unavailable, accelerating accurate analysis of experimental imaging.

Problem

• Enables zero-shot segmentation for scarce scientific images
• Minimizes data readiness barriers with no-code platform
• Improves accuracy for amorphous and crystalline catalyst samples

Innovation

• Lightweight multi-modal adaptation for zero-shot
• Human-in-the-loop refinement for scientific images
• Heuristic-based temporal enhancement options