TracSum: A New Benchmark for Aspect-Based Summarization with Sentence-Level Traceability in Medical Domain

2025-08-19

AI Summary

Medical large language model (LLM) summaries frequently suffer from factual inaccuracies and untraceable information sources, undermining their clinical credibility. To address this, we introduce TracSum—the first benchmark enabling sentence-level traceability for aspect-oriented medical summarization—comprising seven clinical aspects and 3,500 human-annotated summary–citation pairs. We propose a fine-grained evaluation framework and a novel “Track-Then-Sum” generation paradigm that jointly optimizes LLM-based summarization and sentence-level evidence grounding. Experiments demonstrate substantial improvements in factual accuracy and completeness over baseline methods. Human evaluation confirms the benchmark’s reliability and practical utility for clinical applications. TracSum establishes a standardized evaluation platform and a new methodological paradigm for developing verifiable, traceable medical summarization systems.

Technology Category

• Natural Language Processing: Summarization
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Data Visualization & Summarization

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Normalization, clustering, classification, and summarization of Web text

Abstract

While document summarization with LLMs has enhanced access to textual information, concerns about the factual accuracy of these summaries persist, especially in the medical domain. Tracing evidence from which summaries are derived enables users to assess their accuracy, thereby alleviating this concern. In this paper, we introduce TracSum, a novel benchmark for traceable, aspect-based summarization, in which generated summaries are paired with sentence-level citations, enabling users to trace back to the original context. First, we annotate 500 medical abstracts for seven key medical aspects, yielding 3.5K summary-citation pairs. We then propose a fine-grained evaluation framework for this new task, designed to assess the completeness and consistency of generated content using four metrics. Finally, we introduce a summarization pipeline, Track-Then-Sum, which serves as a baseline method for comparison. In experiments, we evaluate both this baseline and a set of LLMs on TracSum, and conduct a human evaluation to assess the evaluation results. The findings demonstrate that TracSum can serve as an effective benchmark for traceable, aspect-based summarization tasks. We also observe that explicitly performing sentence-level tracking prior to summarization enhances generation accuracy, while incorporating the full context further improves completeness.

Problem

• Develops traceable aspect-based summarization benchmark for medical texts
• Addresses factual accuracy concerns in LLM-generated medical summaries
• Enables sentence-level citation tracing to original source documents

Innovation

• Sentence-level traceability with citations
• Aspect-based summarization benchmark creation
• Track-Then-Sum pipeline baseline method