Automated radiotherapy treatment planning guided by GPT-4Vision

2024-06-21

arXiv.org

AI Summary

Radiotherapy treatment planning is time-consuming, highly subjective, and requires iterative trade-offs among conflicting clinical objectives. This paper introduces GPT-RadPlan—a novel automated framework that integrates a multimodal large language model (GPT-4V) as an intelligent agent directly into the clinical radiotherapy planning workflow. Without model fine-tuning, it achieves end-to-end inverse planning optimization via clinical-protocol-driven in-context learning and explicit dosimetric constraint modeling. Its key contribution lies in pioneering zero-shot multimodal reasoning—jointly interpreting medical images and structured clinical protocols—to replace conventional task-specific supervised training. Evaluated on prostate and head-and-neck cancer cases, GPT-RadPlan produces plans satisfying 100% of clinical dosimetric constraints, with superior target coverage and organ-at-risk sparing compared to manual plans—achieving quality at or exceeding expert-level standards.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Radiotherapy treatment planning is a time-consuming and potentially subjective process that requires the iterative adjustment of model parameters to balance multiple conflicting objectives. Recent advancements in large foundation models offer promising avenues for addressing the challenges in planning and clinical decision-making. This study introduces GPT-RadPlan, a fully automated treatment planning framework that harnesses prior radiation oncology knowledge encoded in multi-modal large language models, such as GPT-4Vision (GPT-4V) from OpenAI. GPT-RadPlan is made aware of planning protocols as context and acts as an expert human planner, capable of guiding a treatment planning process. Via in-context learning, we incorporate clinical protocols for various disease sites as prompts to enable GPT-4V to acquire treatment planning domain knowledge. The resulting GPT-RadPlan agent is integrated into our in-house inverse treatment planning system through an API. The efficacy of the automated planning system is showcased using multiple prostate and head & neck cancer cases, where we compared GPT-RadPlan results to clinical plans. In all cases, GPT-RadPlan either outperformed or matched the clinical plans, demonstrating superior target coverage and organ-at-risk sparing. Consistently satisfying the dosimetric objectives in the clinical protocol, GPT-RadPlan represents the first multimodal large language model agent that mimics the behaviors of human planners in radiation oncology clinics, achieving remarkable results in automating the treatment planning process without the need for additional training.

Problem

• Automating radiotherapy planning to reduce time and subjectivity
• Integrating AI with clinical knowledge for better plan optimization
• Improving target coverage and reducing organ-at-risk doses

Innovation

• GPT-4V integrates radiation oncology knowledge
• Automated planning via in-context learning
• API links GPT-RadPlan to treatment system