ε-Seg: Sparsely Supervised Semantic Segmentation of Microscopy Data

2025-10-21

AI Summary

Semantic segmentation of biological electron microscopy (EM) images faces dual challenges: extreme label sparsity (often <0.05%) and intricate structural complexity. To address this, we propose ε-Seg—a weakly supervised segmentation framework built upon a hierarchical variational autoencoder. It innovatively integrates a center-region masking mechanism, an image inpainting loss, and contrastive learning guided by sparse labels, while employing a Gaussian mixture model prior to regularize latent-space clustering. Crucially, ε-Seg replaces conventional post-hoc clustering with a lightweight MLP head that directly regresses pixel-wise semantic labels. Evaluated on two mainstream dense EM benchmarks, ε-Seg achieves performance on par with fully supervised methods using only 0.05% labeled data. Moreover, it demonstrates strong cross-modal generalization to fluorescence microscopy images. These results significantly enhance the practicality and robustness of few-shot segmentation for biological imaging.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Semi-Supervised Learning
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Semantic segmentation of electron microscopy (EM) images of biological samples remains a challenge in the life sciences. EM data captures details of biological structures, sometimes with such complexity that even human observers can find it overwhelming. We introduce ε-Seg, a method based on hierarchical variational autoencoders (HVAEs), employing center-region masking, sparse label contrastive learning (CL), a Gaussian mixture model (GMM) prior, and clustering-free label prediction. Center-region masking and the inpainting loss encourage the model to learn robust and representative embeddings to distinguish the desired classes, even if training labels are sparse (0.05% of the total image data or less). For optimal performance, we employ CL and a GMM prior to shape the latent space of the HVAE such that encoded input patches tend to cluster wrt. the semantic classes we wish to distinguish. Finally, instead of clustering latent embeddings for semantic segmentation, we propose a MLP semantic segmentation head to directly predict class labels from latent embeddings. We show empirical results of ε-Seg and baseline methods on 2 dense EM datasets of biological tissues and demonstrate the applicability of our method also on fluorescence microscopy data. Our results show that ε-Seg is capable of achieving competitive sparsely-supervised segmentation results on complex biological image data, even if only limited amounts of training labels are available.

Problem

• Semantic segmentation of electron microscopy images with sparse labels
• Learning robust embeddings from limited training data for microscopy
• Improving segmentation accuracy on complex biological image datasets

Innovation

• Uses hierarchical variational autoencoders with center-region masking
• Employs contrastive learning and Gaussian mixture prior
• Directly predicts labels via MLP instead of clustering