What the Waveform Knows: Transparent-first Speech and Audio Intelligence with Caption Studio

2026-07-21

AI Summary

This work addresses the lack of transparency and poor interpretability in conventional speech and audio analysis systems by proposing a “transparency-first” framework and implementing an end-to-end speech intelligence platform, Caption Studio. The platform employs a three-tier architecture that integrates Whisper-like automatic speech recognition (ASR), pyannote-based speaker diarization, and multidimensional signal-level analyses—including fundamental frequency, speech rate, filler words, and emotion—while explicitly labeling each metric as “measured,” “derived,” or “unavailable” to enhance traceability and credibility. Built on FastAPI for high-availability enterprise deployment, Caption Studio supports real-time processing, seamless workflow integration, and scalable downstream applications, all while incorporating comprehensive interpretability mechanisms and uncertainty management.

Abstract

Caption Studio is a transparency-first speech and audio intelligence platform that transforms spoken audio and video into structured, searchable content through automated transcription, speaker diarization, speech analytics, signal-level audio analysis, and subtitle generation. The system is built on a FastAPI backend with a real-time dashboard and adopts a three-layer architecture comprising (i) a transcription and diarization core based on Whisper-class automatic speech recognition and pyannote speaker diarization, (ii) an audio intelligence layer that extracts acoustic and linguistic features, including waveforms, spectrograms, pitch, speaking rate, silence, filler-word frequency, and sentiment, directly from the audio signal, and (iii) an integration layer that supports data export and downstream workflow integration. A principal contribution of this work is the transparency-first framework, in which every reported metric is explicitly identified as measured, derived, or unavailable, thereby improving the traceability, interpretability, and reliability of speech analytics. The paper presents the system architecture, benchmarking methodology, explainability and uncertainty framework, and key considerations for enterprise-scale deployment.

Problem

• transparency
• speech analytics
• audio intelligence
• interpretability
• reliability

Innovation

• transparency-first
• audio intelligence
• speaker diarization
• speech analytics
• explainability