Voice Memory for Agentic Speech Recognition

2026-07-28

AI Summary

This work addresses the problem of over-correction in conventional speech recognition systems operating in dynamic domains, which often degrades otherwise accurate transcriptions. To mitigate this, the authors propose a Voice Memory mechanism featuring a frozen corrector that, during inference, consults a domain-specific memory file to determine whether to revise a recognition hypothesis. An asynchronous optimizer updates this memory using a bounded editing strategy, accepting modifications only when they strictly improve the score on a held-out validation set. Built upon a “listener-thinker” decoupled architecture, the approach couples correction and optimization solely through lightweight memory files—without altering model parameters—thereby ensuring auditability, portability, and effective suppression of over-correction. Evaluated across ten HyPoradise domains, the method reduces the weighted word error rate from 8.36% to 7.47% (with three contextual examples), achieves a substantial drop to 3.40% on aviation commands, and reaches 10.46% on CHiME-4, with no observed performance regressions.

Abstract

We present Voice Memory, a inference-only scheme for agentic speech recognition: at stream time, a frozen corrector reads a single per-domain memory.md and decides per utterance whether to act on the hypothesis or abstain and keep the 1-best. Asynchronously, a score-gated optimizer revises that file through bounded edits, accepting an edit only when it strictly improves a held-out score. Extended from classical ASR-LM framework, we refer this split the listener-thinker architecture; the two roles are coupled only through the memory, so no weights change and the learned skill stays auditable and portable. Restraint turns out to be the operative skill this loop discovers: unconstrained generative error correction (GER) over-corrects, breaking correct tokens on up to 64% of its edits on financial news, and Voice Memory, reduces this rate to 35%. Across ten HyPoradise domains with an open corrector, Voice Memory, lowers weighted word error rate from 8.36% to 7.52% (7.47% with three added in-context examples) without regressing any dataset below its 1-best baseline; gains concentrate where recoverable headroom is largest, including air-travel commands (8.40% to 3.40%) and noisy far-field speech (CHiME-4, 12.69% to 10.46%). The memory transfers across corrector families and adds zero parameters to the inference path. A demo and example code are provided for future studies.

Problem

• agentic speech recognition
• voice memory
• error correction
• word error rate
• auditable AI

Innovation

• Voice Memory
• agentic speech recognition
• listener-thinker architecture
• score-gated optimization
• zero-parameter inference