MARL-MambaContour: Unleashing Multi-Agent Deep Reinforcement Learning for Active Contour Optimization in Medical Image Segmentation

2025-06-23

AI Summary

Traditional pixel-wise medical image segmentation suffers from insufficient topological consistency and limited holistic anatomical structure awareness, leading to inaccurate object boundary delineation. To address this, we propose a multi-agent collaborative active contour framework—the first to integrate multi-agent reinforcement learning into active contour modeling. Our method employs a Mamba-based policy network coupled with a Soft Actor-Critic optimizer, introduces a dynamic entropy regularization mechanism (ERAM) to enhance exploration stability, and designs a bidirectional cross-attention hidden-state fusion module (BCHFM) to strengthen long-range inter-agent collaboration and morphological awareness. Evaluated on five benchmark medical imaging datasets, the framework achieves state-of-the-art performance, significantly improving segmentation accuracy for complex boundaries, topological fidelity, and robustness—demonstrating strong potential for clinical deployment.

Technology Category

• Multiagent Systems: Multiagent Learning
• Machine Learning: Active Learning
• Computer Vision: Segmentation

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We introduce MARL-MambaContour, the first contour-based medical image segmentation framework based on Multi-Agent Reinforcement Learning (MARL). Our approach reframes segmentation as a multi-agent cooperation task focused on generate topologically consistent object-level contours, addressing the limitations of traditional pixel-based methods which could lack topological constraints and holistic structural awareness of anatomical regions. Each contour point is modeled as an autonomous agent that iteratively adjusts its position to align precisely with the target boundary, enabling adaptation to blurred edges and intricate morphologies common in medical images. This iterative adjustment process is optimized by a contour-specific Soft Actor-Critic (SAC) algorithm, further enhanced with the Entropy Regularization Adjustment Mechanism (ERAM) which dynamically balance agent exploration with contour smoothness. Furthermore, the framework incorporates a Mamba-based policy network featuring a novel Bidirectional Cross-attention Hidden-state Fusion Mechanism (BCHFM). This mechanism mitigates potential memory confusion limitations associated with long-range modeling in state space models, thereby facilitating more accurate inter-agent information exchange and informed decision-making. Extensive experiments on five diverse medical imaging datasets demonstrate the state-of-the-art performance of MARL-MambaContour, highlighting its potential as an accurate and robust clinical application.

Problem

• Optimizing active contours for medical image segmentation
• Addressing topological constraints in pixel-based segmentation methods
• Enhancing contour accuracy with multi-agent reinforcement learning

Innovation

• Multi-Agent Reinforcement Learning for contour optimization
• Contour-specific SAC algorithm with Entropy Regularization
• Mamba-based policy network with Bidirectional Cross-attention