EndoVLM: An Endoscopy Vision-Language Pre-training Model via Anatomy-Guided Sparsity and Progressive Alignment

2026-08-05

AI Summary

This work addresses the limitations of existing endoscopic foundation models, which predominantly rely on unimodal self-supervised learning and struggle to effectively integrate semantic information from clinical reports or align anatomical descriptions with highly redundant visual streams. To overcome these challenges, we propose a vision–language pretraining framework that leverages anatomical guidance to perform sparse pooling for keyframe extraction across 348K endoscopic examinations. By integrating a progressive semantic alignment strategy with a semantics-focused masked autoencoder, our approach achieves fine-grained cross-modal alignment from the patient level down to lesion-level granularity. This method establishes the first effective and precise vision–language joint modeling in endoscopy, outperforming current foundation models on multiple downstream tasks, matching specialized approaches, and demonstrating strong zero-shot generalization—highlighting its significant potential for clinical deployment.

Abstract

The development of foundation models (FMs) is crucial for advancing endoscopic image analysis. However, existing endoscopy FMs mainly rely on self-supervised learning from uni-modal images or videos, overlooking the rich semantic knowledge contained in clinical reports. Furthermore, effectively leveraging these records is hindered by a fundamental modality gap: structured anatomical descriptions are not naturally mapped to specific frames within the high-redundancy, uncurated visual streams. In this paper, we present EndoVLM, a novel vision-language FM pre-trained on over 348K endoscopic examinations, each pairing a clinical report with its corresponding image collection. An Anatomy-Guided Sparse Pooling mechanism utilizes textual descriptions as queries to drive sparse attention, efficiently aggregating semantically salient frames into anatomy-specific visual representations across redundant image-sets. Next, a Progressive Semantic-Aware Alignment strategy models clinical taxonomy (anatomy and pathological status) via structured soft targets, bridging the gap from global patient-level matching to fine-grained localized alignment. Finally, a Semantic-Concentrated Masked Autoencoder is applied exclusively to these semantic-rich frames, integrating low-level visual precision with robust high-level semantic representation. Extensive experiments across various downstream tasks demonstrate that EndoVLM outperforms existing foundation models and remains competitive with task-specific methods. Remarkably, EndoVLM also exhibits robust zero-shot generalization capabilities, highlighting its potential for broader clinical application.

Problem

• endoscopy
• vision-language
• modality gap
• clinical reports
• anatomical alignment

Innovation

• Anatomy-Guided Sparsity
• Progressive Alignment
• Vision-Language Pre-training
• Semantic-Concentrated Masked Autoencoder
• Endoscopy Foundation Model