duration prediction

Neural text-to-speech (TTS) systems suffer from fragile phoneme–text alignment—especially for long sentences and out-of-domain text—leading to word omissions and repetitions. To address this, we propose a conditional adaptive phoneme duration distribution prediction framework. Unlike conventional approaches assuming fixed duration distributions (e.g., Gaussian), our method is the first to model duration distributions conditioned on textual input, enabling context-aware variability in phoneme timing. Integrated into a non-autoregressive end-to-end TTS architecture, it employs an attention-enhanced external duration predictor. Crucially, the framework demonstrates strong zero-shot robustness to audio–text mismatches—e.g., when prompted with inconsistent or noisy audio references. Experiments show a 11.3% absolute improvement in phoneme-level alignment accuracy, substantially mitigating omissions and repetitions. Moreover, the model exhibits significantly enhanced generalization across unseen domains and sentence structures compared to baseline models.

Addressing the challenges of duration modeling and the trade-off between prosody and speaker identity in text-to-speech (TTS) for low-resource Indian languages, this work systematically investigates explicit duration prediction strategies—namely, speech completion-driven and duration prompt-enhanced approaches—within a non-autoregressive continuous normalizing flow (CNF) framework for single-speaker TTS. Experiments span multiple Indian languages and reveal a clear trade-off: completion-based prediction significantly improves intelligibility, whereas prompt-based strategies better preserve speaker timbre consistency. To our knowledge, this is the first study to empirically validate the effectiveness and interpretability of explicit duration modeling in low-resource multilingual TTS. The results establish a modular, controllable duration modeling paradigm—comprising decoupled, plug-in duration components—that advances resource-constrained TTS system design.

This work addresses the challenge of achieving fine-grained, token-level control over duration and pauses in high-fidelity text-to-speech (TTS) synthesis. It introduces, for the first time, an explicit token-level modeling framework that conditions on both duration and pause information, integrating high-precision duration supervision, a zero-bias correction mechanism, and a robust training strategy that maintains performance even when control signals are absent. The proposed approach enables natural speech synthesis without explicit timing inputs while supporting targeted, local temporal editing when desired. Experimental results demonstrate that the method significantly outperforms autoregressive baselines on duration control benchmarks and achieves low-bias, reproducible fine-grained timing manipulation in practical applications such as navigation prompts, expressive reading, and accessible code narration.

Existing speech translation methods achieve high translation quality but fail to preserve source speech duration, speaker identity, and speaking rate—limiting their applicability in dubbing for film and television. To address this, we propose the first end-to-end textless speech-to-speech cross-lingual dubbing translation framework. Our approach leverages a discrete diffusion model to translate speech into acoustic units, augmented with explicit duration control and textless speaking-rate adaptation—enabling temporally aligned, purely speech-based dubbing for the first time. By combining conditional flow-matching synthesis with unit-level speaking-rate guidance, the generated speech achieves high fidelity to the source in timbre, prosody, and temporal alignment. Experiments demonstrate state-of-the-art translation quality alongside significant improvements in dubbing naturalness and lip-sync accuracy, providing a practical, production-ready solution for audiovisual localization.

This work addresses the challenge faced by resource-constrained teams in developing high-performance text-to-speech (TTS) systems, which typically rely on massive proprietary datasets and complex multi-stage architectures. The authors propose a lightweight autoregressive TTS system featuring an extremely streamlined architecture, rigorous data engineering, and a novel Q-Former-based conditioning mechanism that effectively disentangles speaker identity from expressive style. This enables zero-shot voice cloning as well as synthesis of emotion, paralinguistic cues, and Chinese dialects. Trained exclusively on 200K hours of open-source data using a reproducible multi-stage preprocessing pipeline and cross-sample paired training, the system achieves a word error rate (WER) of 1.50% and character error rate (CER) of 0.87% on the Seed-TTS Eval benchmark for English and Chinese, respectively, with speaker similarity scores of 0.862 and 0.815—outperforming baselines trained on substantially larger datasets.

Existing accent normalization systems often suffer from unnatural outputs or content distortion due to scarce training data and rigid duration modeling. To address these limitations, this work proposes a “source synthesis” training strategy that pairs synthetic non-native speech with real native speech, enabling effective training without any authentic L2 data. Furthermore, we introduce CosyAccent, a non-autoregressive model that integrates implicit prosody modeling with explicit total duration control. This approach effectively mitigates TTS artifacts and, remarkably, outperforms strong baselines trained on real non-native speech—despite using no genuine L2 utterances—achieving significant improvements in both content fidelity and speech naturalness.

Current spoken dialogue models lack temporal awareness, making it difficult to follow duration-related instructions—such as “generate a response lasting approximately 15 seconds”—which degrades interactive user experience. To address this limitation, this work proposes TiCo, a method that endows the model with time awareness by incorporating Spoken Time Markers (STMs) during generation, enabling dynamic adjustment of outputs to meet target durations. TiCo requires only a small amount of real data combined with self-generated samples and does not rely on additional question-answer pairs; it leverages reinforcement learning for efficient training. Experimental results demonstrate that TiCo significantly improves adherence to duration constraints while preserving the naturalness and semantic quality of responses.

This work addresses the fundamental trade-off in real-time spoken dialogue systems between low latency and high semantic quality: end-to-end speech-to-speech (S2S) models offer rapid responses but limited linguistic depth, whereas cascaded ASR+LLM pipelines yield semantically rich outputs at the cost of higher latency. To reconcile this tension, the authors propose a novel dual-path parallel architecture that introduces speculative generation to real-time spoken interaction. Upon detecting a speaker turn, the system concurrently executes a fast path (S2S) to generate an immediately synthesizable response prefix and a slow path (ASR+LLM) to produce high-quality continuation content, dynamically fusing both streams via a lightweight learned verifier. Experiments demonstrate that this approach achieves P90 audio onset latency comparable to pure S2S models while preserving 99% of the average response quality of the cascaded system, with performance gains amplifying as the slow-path model scales.

This study investigates the impact of temporal characteristics—such as pauses and overlaps—in synthetic multi-speaker conversations on automatic speech recognition (ASR) performance. For the first time, conversational timing is explicitly modeled as a controllable training variable by parameterizing pause and overlap distributions using exponential tilting families. Diverse synthetic data are generated in a four-dimensional space through Latin hypercube sampling combined with multi-objective Bayesian optimization to train ASR systems. Results demonstrate that increased speech overlap significantly reduces cpWER, whereas longer and more variable inter-turn gaps degrade performance. Although Bayesian optimization yields only modest gains, the core contribution lies in uncovering the causal mechanisms linking temporal structure to ASR accuracy, underscoring the importance of task-oriented temporal diagnostics over mere corpus similarity in synthetic data design.

This study addresses the limitations of conventional automatic speech recognition (ASR) systems, which are constrained by short-term context modeling and the i.i.d. assumption, hindering effective exploitation of long-range acoustic and linguistic information. For the first time, this work empirically demonstrates that incorporating contextual information spanning up to 21.8 minutes significantly improves recognition performance in end-to-end attention-based ASR models, achieving a relative word error rate reduction of up to 14.2%. Through systematic evaluation of key architectural factors—including positional encoding schemes, model depth, and width—the study elucidates their impact on modeling ultra-long sequences (up to one hour), thereby providing both an effective architecture and empirical foundation for long-context speech recognition.