Transforming Multimodal Models into Action Models for Radiotherapy

2025-02-06

AI Summary

Radiotherapy treatment planning relies on labor-intensive, subjective, and non-standardized manual iterations. To address these limitations, this work proposes the first action-oriented multimodal foundation model paradigm for clinical treatment decision-making. Leveraging few-shot reinforcement learning, it effectively reuses pre-trained physical, anatomical, and radiobiological knowledge—eliminating the need for large-scale annotated data—and enables end-to-end plan optimization. The method integrates a multimodal large model (MLM), Monte Carlo dose simulation, and policy fine-tuning. In prostate cancer simulation experiments, it achieves a 23% improvement in reward score over conventional RL approaches, delivers superior dosimetric outcomes—including +4.2% target coverage and −18.7% organ-at-risk dose—and accelerates planning by 5×. Critically, it enhances clinical interpretability and practical deployability.

Technology Category

• Machine Learning: Multimodal Learning
• Planning, Routing, and Scheduling: Planning with Language Models
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Radiotherapy is a crucial cancer treatment that demands precise planning to balance tumor eradication and preservation of healthy tissue. Traditional treatment planning (TP) is iterative, time-consuming, and reliant on human expertise, which can potentially introduce variability and inefficiency. We propose a novel framework to transform a large multimodal foundation model (MLM) into an action model for TP using a few-shot reinforcement learning (RL) approach. Our method leverages the MLM's extensive pre-existing knowledge of physics, radiation, and anatomy, enhancing it through a few-shot learning process. This allows the model to iteratively improve treatment plans using a Monte Carlo simulator. Our results demonstrate that this method outperforms conventional RL-based approaches in both quality and efficiency, achieving higher reward scores and more optimal dose distributions in simulations on prostate cancer data. This proof-of-concept suggests a promising direction for integrating advanced AI models into clinical workflows, potentially enhancing the speed, quality, and standardization of radiotherapy treatment planning.

Problem

• Transform multimodal models for radiotherapy planning
• Enhance treatment plans using few-shot reinforcement learning
• Improve efficiency and quality of radiotherapy simulations

Innovation

• Transforms multimodal models into action models
• Uses few-shot reinforcement learning approach
• Iteratively improves with Monte Carlo simulator