AgenticASR: Refining Speech Recognition in Real-World Scenarios via an Agentic Approach

2026-07-30

AI Summary

This work addresses the limitations of conventional automatic speech recognition (ASR), whose outputs often contain disfluencies such as fillers, repetitions, and self-corrections, impairing readability and lacking the ability to dynamically revise previously generated text to reflect the speaker’s final intent. To overcome this, the paper introduces the Agentic Speech Recognition (AgenticSR) task and proposes a novel two-stage ASR–Refiner architecture that enables dynamic textual revision in streaming ASR for the first time, supporting continuous output and real-time updates for arbitrarily long audio streams. By integrating context-boundary control, online inference, and in-place text replacement strategies, the system produces normalized, written-style transcripts while preserving semantic intent. The authors also construct AASR-Bench, the first bilingual fine-grained evaluation benchmark for this task. Experiments demonstrate state-of-the-art performance across multiple ASR frontends, with strong agreement between human and AI evaluations, achieving an effective trade-off between transcription quality and latency.

Abstract

Automatic speech recognition (ASR) has achieved substantial gains in transcription accuracy, yet verbatim transcription does not necessarily produce readily usable text. It retains fillers, repetitions, false starts, and self-corrections that increase reading effort, obscure the speaker's final intent, and propagate unresolved or abandoned content to downstream tasks. Existing spoken-to-written methods process completed audio or transcripts but cannot revise emitted text when later speech changes how preceding content should be interpreted. We therefore formulate Agentic Speech Recognition (AgenticSR), an audio-to-clean-text task that removes disfluencies, resolves self-corrections, and normalizes written form while preserving the speaker's final intent. AgenticASR implements this task through an ASR--Refiner architecture that repeatedly transforms a bounded active context and replaces its corresponding output span as audio arrives. This enables continual emission and revision over streams of arbitrary duration. We also introduce AASR-Bench, a bilingual benchmark with fine-grained atomic rubrics. Across multiple ASR front ends, AgenticASR attains the highest AASR-Bench scores among evaluated systems. A human--AI agreement study shows that rubric-based judgments align with independent expert assessments. Ablations characterize Refiner capacity, context length, and the quality--latency trade-off between online and offline inference. Together, these results establish AgenticASR as a practical framework for intent-preserving clean transcription during ongoing speech. Code, AASR-Bench, and a demo will be released at https://github.com/AnXMuy/AgenticASR.

Problem

• disfluency removal
• self-correction resolution
• intent preservation
• spoken-to-written text
• streaming ASR

Innovation

• AgenticASR
• disfluency removal
• incremental refinement
• intent-preserving transcription
• streaming speech recognition