medical image annotation

Clinical multimodal modeling faces persistent challenges including missing modalities, limited sample sizes, dimensional imbalance, and insufficient interpretability. To address these, we conduct the first structured review of 69 medical multimodal studies, establishing a “problem–solution” mapping framework, and propose guidelines for fusion strategy selection and an interpretability evaluation pathway. Methodologically, we integrate transfer learning, generative models, cross-modal attention mechanisms, and neural architecture search—emphasizing modality alignment and adaptive fusion. We distill five major challenge categories and their empirically validated solutions, yielding a comprehensive technical roadmap spanning medical imaging, genomics, wearable sensors, and electronic health records. This work provides both theoretical foundations and practical paradigms for designing, evaluating, and clinically deploying multimodal AI systems in healthcare.

Medical multimodal datasets commonly suffer from scarcity of image–text pairs and insufficient multi-granularity annotations, hindering progress in medical captioning, report generation, and vision tasks. To address this, we introduce the first large-scale medical multimodal dataset encompassing 10 imaging modalities and over 25 million images, with dual-level annotations—global (modality-/organ-level) and local (ROI-, texture-, and region-association-level)—covering 65+ diseases. We propose an automated, human-annotation-free multi-granularity labeling pipeline integrating domain-expert models (for ROI localization), medical knowledge base retrieval, and retrieval-augmented multimodal large language models (MLLMs) to generate image–ROI–description triplets. Leveraging the LLaVA architecture, we conduct both pretraining and fine-tuning on these triplets. Our method yields LLaVA-Tri, which achieves state-of-the-art performance on VQA-RAD, SLAKE, and PathVQA, significantly advancing medical multimodal understanding, radiology report generation, classification, and segmentation.

Medical imaging datasets commonly suffer from label noise, shortcut learning, missing metadata, and challenges in retrospectively addressing newly discovered issues (e.g., biases, artifacts) post-publication—undermining model robustness and clinical reliability. To address these challenges, we propose the first “dynamic living review” paradigm for medical imaging datasets, establishing a full-lifecycle data governance system. We design a structured SQL database and a standardized metadata framework to enable traceable, cross-referenced linkage among datasets, publications, and documented research flaws (e.g., biases, annotation errors, shortcut effects). Additionally, we develop an open-source, web-based interactive knowledge graph to facilitate community-driven verification and iterative curation. The system has archived over 100 documented flaws across multimodal imaging datasets, advancing practical adoption of standardized data documentation, annotation quality assessment, and fairness auditing in medical AI.

Medical image analysis is hindered by data fragmentation, format heterogeneity, and inconsistent annotations, leading to high preprocessing costs and poor model reproducibility. To address these challenges, we introduce MedIMeta—the first standardized, off-the-shelf, multi-domain, multi-task medical imaging metadata set. MedIMeta unifies 19 publicly available datasets across 10 clinical imaging domains and 54 diagnostic tasks. It establishes a novel cross-domain, multi-task metadata paradigm, standardizing data formats, annotation protocols, and evaluation interfaces end-to-end. We further develop a robust data engineering pipeline featuring cross-modal normalization, task-semantic alignment encoding, and native PyTorch encapsulation, alongside benchmarks for both fully supervised and cross-domain few-shot learning. Experiments demonstrate that MedIMeta substantially lowers development barriers: model reproduction efficiency improves by over 3×, while performance remains stable and reproducible across diverse settings.

Expert annotation of chest X-ray images is costly and prone to diagnostic bias. Method: This paper proposes a novel, non-expert crowdsourcing paradigm for rapid annotation of anatomical and device-level visual features (e.g., chest tubes)—rather than diagnostic labels—and introduces NEATX, a new benchmark dataset. It designs a pathology-consistency evaluation framework integrating YOLO and RetinaNet for object detection, and quantifies inter-annotator reliability using Cohen’s and Fleiss’ kappa. The framework yields 4.5k newly annotated catheter instances on NIH-CXR14 and PadChest. Contribution/Results: Detectors trained solely on non-expert annotations generalize robustly to expert-labeled data; inter-annotator agreement reaches moderate to near-perfect levels (κ = 0.40–0.89). This work establishes a reproducible, low-bias, and cost-effective methodology for medical image data curation, accompanied by an empirically validated benchmark.

Medical image segmentation is hindered by the scarcity of high-quality annotated data and the prohibitively high cost of expert annotation. To address this, we propose a crowdsourcing-enhanced framework that synergistically integrates artificial intelligence (AI) and citizen science. Our approach features a cross-modal preprocessing–enabled crowdsourcing annotation platform; AI-assisted initial screening via MedSAM; high-fidelity synthetic data generation using pix2pixGAN; and a novel multi-source label fusion and quality assurance mechanism—establishing an end-to-end “annotate–optimize–synthesize–validate” pipeline. This framework effectively alleviates the small-sample bottleneck: on multimodal medical imaging datasets, it achieves an average 12.3% improvement in Dice coefficient for segmentation and a fivefold increase in annotation efficiency. The proposed paradigm offers a scalable, reproducible solution for training robust segmentation models in low-resource settings.

This work addresses the scarcity of high-quality, large-scale multimodal datasets that jointly support understanding and generation in medical image editing—a key bottleneck hindering the advancement of generative models in this domain. To bridge this gap, the study introduces the first systematic categorization of medical image editing tasks into three types: perception, modification, and transformation. Building upon this framework, the authors construct MieDB-100k, a dataset comprising 100,000 samples generated via modality-specific expert models and rule-driven synthesis, followed by rigorous human validation to ensure clinical fidelity and diversity. Models trained on MieDB-100k demonstrate significantly superior performance and generalization compared to existing open- and closed-source alternatives, establishing a robust foundation for future research in medical image editing.

This work addresses the scarcity of high-quality clinical image–text pairs, a key bottleneck in developing medical multimodal foundation models, as existing resources like PubMed Central often lack fidelity and clinical relevance. The authors propose MedPMC, the first automated and continuously updatable framework that extracts clinically validated image–text pairs from openly licensed literature through a multi-stage pipeline involving figure panel detection, image separation, caption alignment, and medical taxonomy classification. Using this approach, they construct a high-fidelity dataset of 11 million image–text pairs, which substantially enhances model performance: average zero-shot AUC improves by 7.1% across 26 benchmarks, visual question answering accuracy increases by up to 16.9%, and dermatology retrieval Recall@5 rises by 11.7%. The complete framework and dataset are publicly released.

Medical image annotation relies heavily on expert input, which often introduces label noise and ambiguous boundaries that hinder the performance of deep learning models. Addressing this challenge in the context of video capsule endoscopy data, this work proposes the first systematic framework for mislabel detection and cleansing, integrating deep learning with a verification mechanism involving three board-certified gastroenterologists to automatically identify and correct erroneously annotated samples. The proposed approach significantly improves anomaly detection performance, outperforming existing baselines and demonstrating the efficacy and necessity of high-fidelity mislabel correction in medical video analysis.

This work addresses the lack of a unified, machine-verifiable data specification in medical imaging AI, which hinders consistent dataset structure, annotation provenance, quality documentation, and ML-readiness. To bridge this gap, we propose VIDS—an open standard that, for the first time, integrates standardized folder organization, naming conventions, annotation provenance schemas, and quality documentation within a single framework, along with 21 machine-verifiable rules. Built around the NIfTI working format while preserving DICOM metadata, VIDS provides an open-source validator installable via PyPI and supports export to mainstream frameworks such as nnU-Net, MONAI, and COCO. Evaluation reveals that four widely used datasets comply with only 20–39% of VIDS dimensions. We also release LIDC-Hybrid-100, a fully compliant reference dataset comprising 100 CT scans annotated by consensus among four radiologists (mean Dice: 0.7765), which passes all 21 validation checks.

Existing dermoscopic datasets often fall short of clinical requirements in terms of acquisition standardization, metadata completeness, and diagnostic reliability, particularly lacking high-quality data suitable for outpatient and mobile settings in Russia. This work proposes a methodology for constructing a clinically validated dermoscopy dataset by systematically integrating a standardized mobile image acquisition protocol, a 16-field structured metadata schema compatible with ISIC standards, and a three-tier diagnostic verification mechanism comprising clinical annotation, expert consensus, and histopathological confirmation. The resulting dataset comprises 1,026 images spanning nine dermatological conditions, including 39 malignancies all verified by histopathology, thereby substantially enhancing clinical credibility and research utility for tasks such as model evaluation, domain shift analysis, and interpretability studies.