Pathologist Attention-Aligned Report Generation for Prostate Histopathology

2026-07-21

AI Summary

This study addresses the challenge that existing computational pathology report generation models struggle to emulate pathologists’ visual attention to diagnostically critical regions. To bridge this gap, the authors introduce a novel multimodal dataset comprising real-world, multiscale viewport trajectories from practicing pathologists, aligned with their verbal diagnostic descriptions. Leveraging this data, they design an attention alignment loss function to fine-tune a Transformer-based vision-language model, explicitly guiding its image patch attention distribution to match that of human experts. Evaluated on prostate cancer report generation and visual question answering tasks, the proposed method achieves an average improvement of 10.9% in standard NLP metrics and a 19.3% increase in accuracy across five clinically essential report components. Moreover, the generated attention maps demonstrate significantly higher fidelity to actual pathologist gaze patterns.

Abstract

The allocation of visual attention by pathologists during cancer diagnosis is a highly selective process that critically shapes the information extracted from whole-slide images (WSIs). Human attention helps medical imaging tasks such as classification and segmentation, and becomes a strong semantic cue for identifying diagnostically informative regions for report generation. In this paper, we introduce human attention into the training of pathologist report generation models. To this end, we collected a multimodal human-attention dataset of 121 prostate WSIs annotated with pathologists' multi-scale viewport trajectories synchronized with the pathologists' verbal descriptions and cursor movements for five clinically relevant components (e.g., Gleason patterns). Using this dataset, we finetune two report generation models with an attention-alignment loss that regularizes the model attention over image patches to match the distribution of pathologist attention. We evaluate our approach on prostate cancer report generation and visual question answering using two models with different internal attention mechanisms (i.e., how image tokens are integrated into the language decoder). Experiments show average gains of 10.9% on NLP-based metrics and 19.3% in accuracy across five clinically relevant report components. Further, model attention maps extracted at inference time, with minimal computational overhead, align more closely with pathologist attention, providing stronger visual support for the generated reports by highlighting the regions that most influence the output.

Problem

• pathologist attention
• report generation
• prostate histopathology
• whole-slide images
• visual attention alignment

Innovation

• attention alignment
• pathologist attention
• report generation
• whole-slide image
• multimodal dataset