GuideSkill: Evolving Executable LLM Agent Skills for Guideline-Grounded Clinical Reasoning

2026-07-28

AI Summary

This work addresses the challenge that large language models (LLMs) struggle to directly execute diagnostic rules from clinical practice guidelines, typically leveraging guideline texts only indirectly. To bridge this gap, the authors propose GuideSkill, a novel framework that compiles guidelines into executable diagnostic skill functions, establishing a model-agnostic external reasoning layer that outputs ranked diagnostic support scores. The approach comprises two components: zero-shot skills initialized from guidelines (GuideSkill-Zero) and case-driven evolutionary optimization (GuideSkill-Evo), which jointly integrate LLM-based differential diagnosis with skill-based scoring. Evaluated across four benchmarks and four backbone LLMs, GuideSkill-Zero improves macro-accuracy by 13.45% on average over baselines, while GuideSkill-Evo achieves an 18.49% gain over direct reasoning, increases gold-label skill coverage from 56.5% to 99.5%, and surpasses the strongest parameter-finetuned baseline without any backbone model fine-tuning.

Abstract

Clinical practice guidelines (CPGs) encode diagnostic criteria, but LLM systems typically retrieve guideline text or absorb it through training rather than execute its rules. We introduce GuideSkill, an external reasoning layer that compiles disease-specific criteria into executable functions returning ordinal diagnostic-support scores. GuideSkill-Zero is initialized from guidelines, while GuideSkill-Evo uses case--diagnosis pairs to refine covered skills and add missing diagnoses. At inference, an LLM proposes a differential diagnosis, grounds the features required by each matched skill, and fuses its ranking with the executed skill scores. Across four benchmarks and four backbones, GuideSkill-Zero improves macro-average accuracy over guideline RAG by 13.45% on average. GuideSkill-Evo achieves the highest macro-average for every backbone, improves over direct inference by 18.49% relatively, and increases gold-label skill coverage from 56.5% to 99.5%. On Qwen3.5-9B, it also exceeds the strongest parameter-update baseline by 11.16% without updating the backbone. Expert evaluation further indicates that GuideSkill produces clinically sound and broadly acceptable skills, suggesting that its initialized and evolved rules are reliable and practically meaningful. These results support executable skills as a model-agnostic mechanism for combining guideline-derived procedures with case-derived diagnostic patterns.

Problem

• clinical practice guidelines
• executable skills
• clinical reasoning
• diagnostic criteria
• large language models

Innovation

• executable skills
• clinical reasoning
• guideline grounding
• LLM agent
• skill evolution