Improving the Safety and Trustworthiness of Medical AI via Multi-Agent Evaluation Loops

2026-01-19

AI Summary

This work addresses the ethical compliance and safety risks posed by large language models in clinical applications. To this end, the authors propose a scalable, regulation-aligned, and cost-efficient multi-agent iterative alignment framework that orchestrates generative models (DeepSeek R1 and Med-PaLM) with evaluation agents (LLaMA 3.1 and Phi-4) to iteratively refine outputs according to the American Medical Association (AMA) ethical guidelines and a five-tier Safety Risk Assessment protocol (SRA-5). By establishing a closed-loop evaluation mechanism, the approach significantly enhances the safety and trustworthiness of medical AI, achieving an 89% reduction in ethical violations and a 92% risk downgrading rate. DeepSeek R1 demonstrates faster convergence, while Med-PaLM exhibits superior performance in privacy-sensitive scenarios.

Technology Category

• Natural Language Processing: Safety and Robustness
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large Language Models (LLMs) are increasingly applied in healthcare, yet ensuring their ethical integrity and safety compliance remains a major barrier to clinical deployment. This work introduces a multi-agent refinement framework designed to enhance the safety and reliability of medical LLMs through structured, iterative alignment. Our system combines two generative models - DeepSeek R1 and Med-PaLM - with two evaluation agents, LLaMA 3.1 and Phi-4, which assess responses using the American Medical Association's (AMA) Principles of Medical Ethics and a five-tier Safety Risk Assessment (SRA-5) protocol. We evaluate performance across 900 clinically diverse queries spanning nine ethical domains, measuring convergence efficiency, ethical violation reduction, and domain-specific risk behavior. Results demonstrate that DeepSeek R1 achieves faster convergence (mean 2.34 vs. 2.67 iterations), while Med-PaLM shows superior handling of privacy-sensitive scenarios. The iterative multi-agent loop achieved an 89% reduction in ethical violations and a 92% risk downgrade rate, underscoring the effectiveness of our approach. This study presents a scalable, regulator-aligned, and cost-efficient paradigm for governing medical AI safety.

Problem

• Medical AI
• Safety
• Ethical Integrity
• Clinical Deployment
• Large Language Models

Innovation

• multi-agent evaluation
• medical AI safety
• iterative alignment
• ethical compliance
• risk assessment