audio segmentation

To address GPU memory constraints and inefficient batching caused by variable-length inputs in industrial-scale long-audio automatic speech recognition (ASR), this paper proposes a mask-based chunked batching mechanism and a relative right-context modeling approach tailored for the Conformer architecture. It enables, for the first time, padding-free dynamic batching and end-to-end modeling of hour-long speech. Our method supports transcription of audio up to 16 hours on an 80 GB GPU—1.5× longer than FastConformer’s maximum. It reduces word error rate (WER) by 7.7 percentage points absolutely, cuts batch memory consumption and latency by over 3×, and matches the original Conformer’s accuracy on short utterances. The core innovations lie in the synergistic optimization of chunked self-attention, relative positional encoding, and masked chunking—effectively mitigating padding redundancy and extending Conformer’s capability to ultra-long speech with minimal memory overhead.

This work addresses the incompatibility between continuous audio streams on edge devices and the discrete batch requirements of contrastive learning, as well as the challenge of balancing accuracy, latency, and bandwidth under dynamic resource constraints. To this end, the authors propose a streaming contrastive learning framework tailored for heterogeneous ARM-based edge platforms. The framework decouples representation quality from local batch size through distribution modeling, integrates an uncertainty-guided adaptive computation partitioning strategy with a lightweight reinforcement learning mechanism to enable runtime optimization of the accuracy–latency trade-off, and introduces a hybrid loss function to support high-quality representation learning under sparse updates. Experiments across devices ranging from Raspberry Pi 4 to Apple M2 demonstrate up to a 4.7× reduction in per-sample latency, 77.1% less bandwidth usage, 52.3% lower energy consumption, and accuracy degradation within 2.2%.

This work addresses the challenge of applying autoregressive generative models to low-latency streaming target speaker extraction, a task hindered by their inherent reliance on global context. The study proposes a novel chunk-wise interleaved concatenation paradigm to enable efficient real-time inference and introduces a historical context refinement mechanism to mitigate discontinuities at speech segment boundaries. Evaluated on the Libri2Mix dataset, the system achieves 100% inference stability and high speech intelligibility, matching or even surpassing offline baselines in streaming performance. Notably, it attains a real-time factor (RTF) of 0.248 on consumer-grade GPUs, demonstrating practical feasibility for real-world deployment.

Discrete-token speech generation faces a fundamental trade-off between streaming capability and audio quality, primarily due to the high latency induced by global self-attention in autoregressive models. Method: This paper proposes an efficient streaming decoding framework based on flow matching. Its core innovation is a block-wise guided attention masking mechanism that partitions sequences and enforces hierarchical local attention constraints to explicitly model historical dependencies—thereby avoiding the latency overhead of full-sequence attention. The method integrates diffusion Transformers with flow matching to jointly optimize low-latency inference and high-fidelity waveform reconstruction. Contribution/Results: Experiments demonstrate a first-token latency of only 180 ms, significantly accelerated inference speed for long sequences, and speech quality (MOS) on par with non-streaming baselines. The framework enables real-time, interactive speech synthesis without compromising perceptual fidelity.

Large Audio-Language Models (LALMs) face dual challenges in long-audio processing: quadratic attention complexity (O(N²)) and weak temporal dependency modeling; existing benchmarks predominantly target short audio, lacking evaluation capability for long-context understanding. This work introduces the first comprehensive benchmark for long-duration audio understanding, spanning speech, sound, and music domains, supporting input lengths of 2,250–7,500 audio tokens. It systematically establishes three novel evaluation dimensions: long-sequence comprehension, cross-domain generalization, and multi-hop reasoning. Leveraging optimization techniques—including token pruning and KV cache eviction—we conduct joint performance-efficiency evaluation across state-of-the-art LALMs. Results expose critical bottlenecks: sharp accuracy degradation and severe memory overhead under long-sequence inference. The benchmark provides a reproducible evaluation framework and concrete architectural guidance for developing efficient, scalable audio foundation models.

This work addresses the challenge of real-time streaming audiovisual character generation by simultaneously ensuring speech–text alignment, cross-segment visual consistency, and low-latency constraints. The authors propose a decoupled architecture comprising an LLM-driven coordinator that produces frame-level aligned audio conditions and employs a progress-aware pointer to maintain text–speech synchronization. A joint audiovisual DiT model performs localized bidirectional denoising within short temporal windows, augmented with a sink-token memory mechanism to suppress visual drift. Efficient deployment is achieved through a two-stage distillation strategy. Evaluated on a single H100 GPU, the method achieves real-time performance and outperforms existing baselines in text fidelity, audiovisual synchronization, visual quality, and streaming stability.

This work addresses the challenge of flexibly balancing algorithmic latency and enhancement quality in streaming speech enhancement, where existing approaches are largely confined to binary choices between causal and non-causal processing. The authors propose a unified framework supporting continuously adjustable latency through asymmetric temporal padding, a dual-buffer mechanism comprising state and look-ahead buffers, and selective state updates. This design enables consistent multi-latency configurations during both training and inference within a single model. Under a strict 1.37M parameter budget, the proposed model achieves latencies ranging from 12.5 to 75.0 ms with a PESQ score of 3.43—outperforming the baseline by 0.08—and demonstrates superior performance at the 12.5 ms causal setting compared to current state-of-the-art causal models.

This work addresses the trade-off between speech perceptual quality and intelligibility in streaming target speaker extraction, where enhancing quality often degrades intelligibility. To mitigate this, the authors propose a Direct Preference Optimization (DPO) method leveraging deep acoustic features from WavLM, introducing for the first time cosine similarity in deep feature space as an anchor for constructing preference pairs. Combined with an enlarged Conformer convolution kernel, this approach enables joint modeling of speech content and speaker identity, effectively preventing reward hacking. Evaluated under a 560 ms streaming chunk setting, the method reduces word error rate from 0.138 to 0.123 (a relative improvement of 10.9%) while slightly improving audio quality and speaker similarity, thereby significantly alleviating the quality–intelligibility trade-off.

This work proposes LATTE, a neural audio codec that addresses the limitations of existing frame-level sequential tokenization in modeling and manipulating global speech attributes—such as speaker identity and background noise—without task-specific supervision. By introducing learnable latent tokens preceding the audio features and quantizing only these non-temporally-aligned tokens for decoding, LATTE constructs a compact, discrete bottleneck representation. This design enables, for the first time, global editing in the audio token space through simple latent token swapping, facilitating unsupervised speaker conversion and denoising. Experiments demonstrate that LATTE achieves high-fidelity audio reconstruction at low bitrates while enabling efficient, controllable manipulation of global speech characteristics without any labeled data or auxiliary supervision.

This work addresses the challenge of achieving both high accuracy and low latency in long-form automatic speech recognition (ASR). The authors propose MURMUR, a novel system that integrates an adaptive block-size chunked pipeline with a sliding-window key-value (KV) cache eviction mechanism to optimize inference efficiency at both inter-chunk and intra-chunk levels. The former eliminates reliance on heuristic boundary alignment common in conventional approaches, while the latter exploits attention sparsity to reduce redundant computation. MURMUR maintains single-pass inference accuracy while substantially lowering latency. On the AMI-IHM dataset, it achieves accuracy comparable to single-pass inference with a 4.2× reduction in latency; when combined with token pruning, it incurs less than 1% relative degradation in tcpWER.