🤖 AI Summary
Current medical research agents lack domain-specific evaluation mechanisms that rigorously assess scientific validity, methodological soundness, reproducibility, and boundary safety. This work proposes MedSkillAudit—the first skill auditing framework tailored for medical research agents—which employs a hierarchical, structured pipeline to evaluate skill readiness prior to deployment. The framework incorporates expert double-blind scoring (0–100), tiered release recommendations, and high-risk flags, and quantifies agreement between the system and human experts using ICC(2,1) and weighted Cohen’s kappa. Evaluated on 75 skills, the system achieved an ICC of 0.449, surpassing inter-human rater agreement (ICC = 0.300) and demonstrating closer alignment with consensus scores (SD = 9.5 vs. 12.4), thereby validating its effectiveness and reliability.
📝 Abstract
Background: Agent skills are increasingly deployed as modular, reusable capability units in AI agent systems. Medical research agent skills require safeguards beyond general-purpose evaluation, including scientific integrity, methodological validity, reproducibility, and boundary safety. This study developed and preliminarily evaluated a domain-specific audit framework for medical research agent skills, with a focus on reliability against expert review. Methods: We developed MedSkillAudit (skill-auditor@1.0), a layered framework assessing skill release readiness before deployment. We evaluated 75 skills across five medical research categories (15 per category). Two experts independently assigned a quality score (0-100), an ordinal release disposition (Production Ready / Limited Release / Beta Only / Reject), and a high-risk failure flag. System-expert agreement was quantified using ICC(2,1) and linearly weighted Cohen's kappa, benchmarked against the human inter-rater baseline. Results: The mean consensus quality score was 72.4 (SD = 13.0); 57.3% of skills fell below the Limited Release threshold. MedSkillAudit achieved ICC(2,1) = 0.449 (95% CI: 0.250-0.610), exceeding the human inter-rater ICC of 0.300. System-consensus score divergence (SD = 9.5) was smaller than inter-expert divergence (SD = 12.4), with no directional bias (Wilcoxon p = 0.613). Protocol Design showed the strongest category-level agreement (ICC = 0.551); Academic Writing showed a negative ICC (-0.567), reflecting a structural rubric-expert mismatch. Conclusions: Domain-specific pre-deployment audit may provide a practical foundation for governing medical research agent skills, complementing general-purpose quality checks with structured audit workflows tailored to scientific use cases.