Supervised Fine-tuning with Synthetic Rationale Data Hurts Real-World Disease Prediction

2026-06-08

AI Summary

This study investigates whether supervised fine-tuning with synthetic chain-of-thought rationales genuinely enhances the clinical predictive performance of language models in a five-year Alzheimer’s disease and related dementias (ADRD) prediction task. Through a large-scale controlled experiment encompassing 504 configurations—including diverse model architectures, dataset scales, and reasoning-oriented base models—augmented by expert evaluation, the work systematically compares rationale-based fine-tuning against label-only fine-tuning. It reveals, for the first time, that despite being medically accurate and well-supported, synthetic rationales consistently and significantly degrade prediction performance during training; yet, when used as few-shot exemplars at inference time, the same rationales improve performance. This finding uncovers a structural conflict between narrative plausibility and discriminative optimization, challenging prevailing assumptions by demonstrating that performance degradation stems from the training objective rather than rationale quality.

Abstract

Supervised fine-tuning with synthetic rationale data is widely assumed to improve language model performance on clinical prediction tasks by teaching models not just what to predict but why. We test this assumption on five-year Alzheimer's disease and related dementias (ADRD) prediction from longitudinal health histories. Across a large-scale controlled experiment of 504 configurations, we find that rationale-based SFT consistently and substantially hurts prediction performance relative to label-only fine-tuning. The degradation persists across model families and data scales, and is not resolved by using a reasoning-oriented base model. Crucially, the failure is not explained by poor rationale quality: human expert annotation confirms that the generated rationales are medically accurate and faithfully grounded in patient-specific evidence, and few-shot experiments show that the same rationales improve performance when used as inference-time demonstrations rather than training targets. We identify the root cause as a structural conflict between narrative plausibility and discriminative optimization. We hope our work paves the path toward a more precise understanding of when and how rationale-based supervision helps and when it does not, guiding the responsible development of language models for high-stakes clinical prediction.

Problem

• supervised fine-tuning
• synthetic rationale
• clinical prediction
• Alzheimer's disease
• language models

Innovation

• supervised fine-tuning
• synthetic rationale
• clinical prediction
• Alzheimer's disease
• reasoning conflict