Model selection meets clinical semantics: Optimizing ICD-10-CM prediction via LLM-as-Judge evaluation, redundancy-aware sampling, and section-aware fine-tuning

2025-09-23

AI Summary

This study addresses three critical challenges in ICD-10-CM coding automation: suboptimal large language model (LLM) selection, insufficient clinical context modeling, and training data redundancy. To tackle these, we propose a modular medical coding framework. First, we introduce an LLM-as-Judge evaluation paradigm integrated with Plackett–Luce ranking for principled, task-specific LLM selection. Second, we design a semantic-similarity-driven redundancy-aware sampling strategy to mitigate data duplication. Third, we develop a section-aware contextual modeling approach coupled with context-sensitive prompting to enhance clinical paragraph understanding and coding consistency. Evaluated on two real-world institutional datasets, our selected open-source LLM—after fine-tuning—significantly outperforms mainstream baselines in both in-domain and out-of-domain settings (F1 gains of 3.2–5.8 percentage points). Further incorporating clinical paragraph enhancement yields additional performance improvements.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Accurate International Classification of Diseases (ICD) coding is critical for clinical documentation, billing, and healthcare analytics, yet it remains a labour-intensive and error-prone task. Although large language models (LLMs) show promise in automating ICD coding, their challenges in base model selection, input contextualization, and training data redundancy limit their effectiveness. We propose a modular framework for ICD-10 Clinical Modification (ICD-10-CM) code prediction that addresses these challenges through principled model selection, redundancy-aware data sampling, and structured input design. The framework integrates an LLM-as-judge evaluation protocol with Plackett-Luce aggregation to assess and rank open-source LLMs based on their intrinsic comprehension of ICD-10-CM code definitions. We introduced embedding-based similarity measures, a redundancy-aware sampling strategy to remove semantically duplicated discharge summaries. We leverage structured discharge summaries from Taiwanese hospitals to evaluate contextual effects and examine section-wise content inclusion under universal and section-specific modelling paradigms. Experiments across two institutional datasets demonstrate that the selected base model after fine-tuning consistently outperforms baseline LLMs in internal and external evaluations. Incorporating more clinical sections consistently improves prediction performance. This study uses open-source LLMs to establish a practical and principled approach to ICD-10-CM code prediction. The proposed framework provides a scalable, institution-ready solution for real-world deployment of automated medical coding systems by combining informed model selection, efficient data refinement, and context-aware prompting.

Problem

• Automating ICD-10-CM coding using LLMs faces model selection challenges
• Training data redundancy limits effectiveness of automated coding systems
• Input contextualization affects prediction accuracy in clinical documentation

Innovation

• LLM-as-Judge evaluation with Plackett-Luce aggregation
• Redundancy-aware sampling to remove duplicate summaries
• Section-aware fine-tuning using structured clinical sections