Beyond English benchmarks: clinical llm evaluation in Brazilian Portuguese

2026-06-05

AI Summary

This study addresses the lack of systematic evaluation of clinical large language models (LLMs) in non-English contexts, which hinders equitable global deployment of medical AI. To bridge this gap, the authors introduce ClinicalBr, the first bilingual (Portuguese–English) clinical decision-making benchmark based on real Brazilian patient cases, spanning 18 specialties and comprising 2,892 parallel cases. The benchmark supports four tasks: diagnosis retrieval, differential diagnosis, test recommendation, and treatment planning. Evaluations across models including MedGemma-27B, Sabiá-4, DeepSeek-R1, and o3-mini reveal that performance disparities between languages are task-dependent: English significantly outperforms Portuguese only in diagnosis retrieval (+7.5–12.1% accuracy), with no significant differences in other tasks. Test recommendation proves most challenging (F1 < 0.10), while cases involving Brazilian endemic diseases are comparatively easier—challenging prevailing assumptions about cross-lingual transfer difficulty.

Abstract

Large Language Models are transforming the support for clinical decision and their application in real scenarios. Yet, most benchmarks are conducted in English, and cross-lingual evaluation is needed to tackle the language gaps in global access. We introduce ClinicalBr, the first bilingual benchmark for clinical decision built from real Brazilian case reports. The corpus contains 2,892 cases drawn from 28 SciELO medical journals, spanning 18 specialties, and is structured as parallel Portuguese-English pairs. Each case supports four evaluation tasks: diagnosis retrieval, differential diagnosis, exam recommendation, and treatment planning. We evaluate four models: MedGemma-27B, Sabiá-4, DeepSeek-R1, and o3-mini, across both languages. The central finding is that the Portuguese-English performance gap is task-dependent, not general. In diagnosis retrieval, English yields a consistent advantage across all models, with +7.5-12.1 accuracy points. This advantage disappears in differential diagnosis, exam recommendation, and treatment planning, where confidence intervals cross zero for most models and Portuguese completeness scores are marginally higher. Brazilian-endemic conditions proved easier than the full corpus, not harder, indicating that tropical presentations are adequately represented in current pre-training. Exam recommendation was the hardest task across all models and both languages, with F1 scores below 0.10, well below the differential diagnosis ceiling of 0.20-0.27.

Problem

• clinical LLM evaluation
• cross-lingual benchmark
• Brazilian Portuguese
• language gap
• non-English medical NLP

Innovation

• ClinicalBr
• bilingual benchmark
• cross-lingual evaluation
• clinical LLM
• Brazilian Portuguese