Robust Summarization of Doctor-Patient Conversations: TalTech Systems for the Beyond Transcription Challenge

2026-07-19

AI Summary

This work addresses the end-to-end task of generating SOAP notes directly from long-duration doctor–patient conversation recordings without relying on intermediate transcription. It proposes a speech summarization approach built upon the Voxtral Mini/Small speech foundation models, enhanced through LoRA-based supervised fine-tuning and a novel DAPO reinforcement learning strategy that incorporates a medical concept matching reward for the first time. Specifically, Open Medical Concept F1 is innovatively employed as the reinforcement learning reward signal, and the study validates the cross-modal transfer efficacy of text-based fine-tuning to speech inputs. The method achieved top performance in both lightweight and heavyweight tracks of the BeTraC challenge, with independent evaluation confirming the lowest hallucination rate and robust generalization on out-of-domain real-world recordings.

Abstract

This paper describes TalTech's submissions to the Beyond Transcription Challenge (BeTraC), which requires generating SOAP notes directly from long doctor-patient conversation recordings, without intermediate transcription. After screening open-weight speech LLMs for long-audio robustness, we adapted Voxtral Mini (lightweight track) and Voxtral Small (heavyweight track) with LoRA supervised fine-tuning followed by DAPO reinforcement learning that uses the challenge metric, Open Medical Concept F1, as its reward. Our systems ranked first in both tracks, and an independent LLM-as-a-judge evaluation showed the lowest hallucination rate among all submissions, indicating that reinforcement learning against a concept-matching metric need not compromise factual reliability. We also find that fine-tuning on text transcripts transfers well to speech input and appears to improve robustness on out-of-domain real recordings.

Problem

• doctor-patient conversations
• SOAP notes
• speech summarization
• robustness
• medical documentation

Innovation

• speech LLM
• reinforcement learning
• SOAP notes
• hallucination reduction
• zero-transcription summarization