Evaluating The Performance of Using Large Language Models to Automate Summarization of CT Simulation Orders in Radiation Oncology

2025-01-27

AI Summary

Manual curation of CT simulation orders in radiation oncology is inefficient and labor-intensive. Method: This study proposes a clinical-scenario-specific large language model (LLM) summarization method, built upon a locally deployed Llama 3.1-405B foundation model. We introduce “order-grouping prompt engineering,” customizing prompts for seven treatment modalities and anatomical sites to enable consistent, cross-modal and cross-diagnostic structured summarization. Instruction templates were co-designed with clinical radiation therapists and validated via human annotation and double-blind review. Results: The LLM achieves 98% clinical accuracy in summarization, with significantly superior formatting consistency and readability compared to manual summaries; performance remains stable across all categories. This approach substantially accelerates CT simulation order processing and establishes a reusable technical paradigm for intelligent radiotherapy workflow automation.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Summarization
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work

Abstract

Purpose: This study aims to use a large language model (LLM) to automate the generation of summaries from the CT simulation orders and evaluate its performance. Materials and Methods: A total of 607 CT simulation orders for patients were collected from the Aria database at our institution. A locally hosted Llama 3.1 405B model, accessed via the Application Programming Interface (API) service, was used to extract keywords from the CT simulation orders and generate summaries. The downloaded CT simulation orders were categorized into seven groups based on treatment modalities and disease sites. For each group, a customized instruction prompt was developed collaboratively with therapists to guide the Llama 3.1 405B model in generating summaries. The ground truth for the corresponding summaries was manually derived by carefully reviewing each CT simulation order and subsequently verified by therapists. The accuracy of the LLM-generated summaries was evaluated by therapists using the verified ground truth as a reference. Results: About 98% of the LLM-generated summaries aligned with the manually generated ground truth in terms of accuracy. Our evaluations showed an improved consistency in format and enhanced readability of the LLM-generated summaries compared to the corresponding therapists-generated summaries. This automated approach demonstrated a consistent performance across all groups, regardless of modality or disease site. Conclusions: This study demonstrated the high precision and consistency of the Llama 3.1 405B model in extracting keywords and summarizing CT simulation orders, suggesting that LLMs have great potential to help with this task, reduce the workload of therapists and improve workflow efficiency.

Problem

• Automated Organization
• Cancer Treatment
• CT Examination Sheets

Innovation

• Llama 3.1 405B Language Model
• Automated CT Summary Generation
• High Accuracy and Clarity in Oncology