SynthEHR-Eviction: Enhancing Eviction SDoH Detection with LLM-Augmented Synthetic EHR Data

2025-07-10

AI Summary

This study addresses the lack of structured coding for eviction—a critical yet underexplored social determinant of health (SDoH)—in electronic health records (EHRs). We propose an LLM-driven detection framework integrating human-in-the-loop collaborative annotation, automated prompt optimization (APO), and LLM-enhanced synthetic data generation. This yields the largest publicly available eviction-related SDoH dataset to date, featuring 14 fine-grained categories and reducing annotation costs by over 80%. Fine-tuning Qwen2.5 and LLaMA3 on this dataset achieves Macro-F1 scores of 88.8% (eviction) and 90.3% (other SDoH) on a human-validated test set—substantially outperforming GPT-4o-APO and BioBERT. Our key contributions are: (1) the first fine-grained eviction annotation taxonomy; (2) a low-cost, high-quality SDoH data curation paradigm; and (3) a model-scale-transferable, LLM-augmented detection methodology for SDoH extraction from clinical text.

Technology Category

• Humans and AI: Human-in-the-loop Machine Learning
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Eviction is a significant yet understudied social determinants of health (SDoH), linked to housing instability, unemployment, and mental health. While eviction appears in unstructured electronic health records (EHRs), it is rarely coded in structured fields, limiting downstream applications. We introduce SynthEHR-Eviction, a scalable pipeline combining LLMs, human-in-the-loop annotation, and automated prompt optimization (APO) to extract eviction statuses from clinical notes. Using this pipeline, we created the largest public eviction-related SDoH dataset to date, comprising 14 fine-grained categories. Fine-tuned LLMs (e.g., Qwen2.5, LLaMA3) trained on SynthEHR-Eviction achieved Macro-F1 scores of 88.8% (eviction) and 90.3% (other SDoH) on human validated data, outperforming GPT-4o-APO (87.8%, 87.3%), GPT-4o-mini-APO (69.1%, 78.1%), and BioBERT (60.7%, 68.3%), while enabling cost-effective deployment across various model sizes. The pipeline reduces annotation effort by over 80%, accelerates dataset creation, enables scalable eviction detection, and generalizes to other information extraction tasks.

Problem

• Detecting eviction status from unstructured EHR data
• Creating scalable synthetic EHR datasets for SDoH
• Reducing annotation effort for eviction-related SDoH extraction

Innovation

• LLM-augmented synthetic EHR data generation
• Human-in-the-loop annotation integration
• Automated prompt optimization for extraction