QwenVLConnector: A Fast, Unified Medical VLM Chatbot for Fine-Grained Clinical Perception and Text Generation

2026-09-19

AI Summary

该研究解决了医疗视觉-语言模型在细粒度临床感知和文本生成上支持不足的问题，通过引入QwenVLConnector，一个基于Qwen2.5-VL的统一聊天机器人框架来解决。

Abstract

Most medical vision-language models (VLMs) excel at open-ended report generation and VQA but provide limited support for structured, fine-grained clinical perception within a unified interface. We present QwenVLConnector, a Qwen2.5-VL-based medical chatbot that unifies classification, multi-label classification, textualized detection, counting, regression, and free-form report generation under a single next-token objective. Our key component is a lightweight dense multi-layer Connector that aggregates low- and high-level visual features, aligns them through the pretrained vision Merger, and fuses them with the final visual representation without increasing sequence length. This design enriches visual tokens with complementary spatial and semantic cues while preserving efficiency. On FLARE-2D, QwenVLConnector improves detection F1 from 0.55 to 0.85, raises single-label classification from 0.37 to 0.51, and boosts report-generation GREEN by up to 18.3 points over the Qwen2.5-VL baseline. We further explore multimodal in-context learning for report generation, showing additional improvements without updating model parameters. Overall, QwenVLConnector offers a unified and efficient framework for combining structured medical perception with open-ended clinical text generation. Our code can be found at https://github.com/plnguyen2908/QwenConnector.

Problem

• Medical VLMs
• Clinical Perception
• Unified Interface
• Structured Tasks
• Report Generation

Innovation

• Qwen2.5-VL
• lightweight dense multi-layer Connector
• visual feature fusion
• unified interface for medical tasks
• improved detection and classification