voice activity detection

This work proposes a knowledge-driven, self-supervised approach to audio segmentation and source separation that circumvents the reliance on large-scale manually annotated data. By integrating external prior knowledge—such as musical scores—into the audio processing pipeline for the first time, the method leverages hidden Markov models to achieve effective segmentation and separation of music and film audio without requiring labeled training data. Evaluated on synthetic datasets, the approach demonstrates strong performance, and in real-world film soundtrack tests, it significantly outperforms purely data-driven methods when incorporating sound-class priors. This represents a notable advance toward annotation-free audio analysis through principled integration of domain knowledge.

This work addresses the challenge of jointly modeling multiple non-lexical acoustic-semantic styles (e.g., speaker identity, emotion) in unsupervised speech segmentation. We propose a general unsupervised segmentation framework based on Speech Language Models (SLMs), the first to leverage SLMs for this task. Our method extracts robust acoustic-semantic joint representations via an SLM and integrates them with unsupervised boundary detection and representation clustering to achieve fine-grained, style-aware segmentation—without requiring any text transcription. Unlike conventional approaches that model only a single style, our framework enables multi-style joint modeling. Experiments demonstrate significant improvements over state-of-the-art baselines across key metrics: boundary detection accuracy, segment purity, and over-segmentation rate. These results validate both the effectiveness of our approach and its strong cross-style generalization capability.

Traditional two-stage audio source separation approaches—first detecting sound events then separating sources—struggle in complex acoustic mixtures due to insufficient fine-grained temporal modeling. To address this, we propose a time-varying collaborative framework: (1) a fine-tuned pre-trained Transformer performs high-accuracy, frame-level sound event detection (SED) to generate dynamic temporal guidance signals; (2) an iterative refinement separation network explicitly models temporal dynamics by jointly incorporating label-conditioned constraints and recursive output feedback. Evaluated on DCASE 2025 Task 4, our method achieves second place, significantly improving both SED F1-score (+2.1%) and separation quality (SI-SNRi +1.8 dB) over strong baselines. Results demonstrate that time-aware collaborative modeling effectively bridges the gap between detection and separation, enabling more accurate and temporally coherent joint optimization.

To address performance degradation in sound event detection (SED) caused by overlapping acoustic events and background noise, this paper proposes a text-query-based SED framework (TQ-SED). First, we introduce AudioSep-DP—a language-driven, end-to-end differentiable audio separation model—by augmenting AudioSep with a dual-path RNN module to enhance dynamic audio modeling. Second, we design a multi-branch specialized detection head to independently identify events from each separated source. To the best of our knowledge, this is the first work to jointly model cross-modal language–audio alignment, a dual-path RNN–CNN hybrid separation architecture, and event detection. Evaluated on the DCASE 2024 Task 9 objective single-model track, TQ-SED achieves first place, improving F1 score by 7.22% over conventional SED methods. The code and pretrained models are publicly available.

Current audio separation models suffer from domain specificity (e.g., speech- or music-only) or reliance on single-modal prompts (e.g., text only), lacking unified support for multimodal prompts—including text, visual masks, and temporal spans. To address this, we propose the first general-purpose, multimodal-controllable audio separation foundation model, built upon a diffusion Transformer architecture. Our approach introduces flow matching-based training, cross-domain large-scale pretraining, and a novel multimodal prompt fusion mechanism. We further construct the first real-world multimodal-annotated benchmark for audio separation and design a reference-free, human-perception-aware evaluation model. Experiments demonstrate that our model achieves state-of-the-art performance across diverse benchmarks—including general sound, speech, music, and instrument separation—significantly outperforming both domain-specific and existing general-purpose methods. This enables truly flexible, controllable separation in open acoustic scenarios.

This work addresses the challenge of localizing partially forged segments in deepfake speech detection by proposing a decoupled two-stage approach. First, a boundary detector identifies acoustic transition points to segment the input utterance into acoustically homogeneous units; then, each segment is independently classified as genuine or spoofed. The method introduces an innovative reflection-based multi-length training strategy, combined with multi-feature extractor fusion and tailored data augmentation, substantially enhancing model robustness and generalization. Evaluated on the PartialSpoof and Half-Truth datasets, the proposed framework achieves state-of-the-art performance, significantly improving both detection accuracy and localization precision of forged regions.

Existing evaluation metrics for spatial soundscapes semantic segmentation (S5), particularly class-aware signal-to-distortion ratio (CA-SDR), fail to disentangle joint errors from audio source separation and sound event classification—especially under pure classification errors. To address this, we propose class-agnostic signal-to-distortion ratio (CI-SDR), a novel metric framework that introduces cross-contamination-aware error penalization without relying on class priors, thereby distinguishing separation fidelity from annotation bias. Through systematic analysis of CI-SDR’s response to typical degradation scenarios—including label noise and source overlap—we demonstrate its sensitivity and orthogonality to both separation and classification errors. Experiments across diverse S5 systems show that CI-SDR significantly improves the accuracy, robustness, and interpretability of inter-system performance comparisons. This work establishes a principled, unified evaluation paradigm for S5, advancing fair and transparent benchmarking.

This work addresses the limitations of existing audio segmentation approaches, which heavily rely on textual transcriptions while neglecting intrinsic audio signals, the impact of ASR errors, and transcription-free evaluation protocols. To overcome these issues, the authors propose AudioSeg, a purely audio-based segmentation model, and conduct a systematic comparison among text-based models, acoustic features, AudioSeg, and multimodal large language models (MLLMs). They further introduce a novel transcription-free evaluation framework based on temporal alignment. Experimental results demonstrate that AudioSeg significantly outperforms text-dependent methods, with silent pauses emerging as the strongest acoustic cue. Although MLLMs are constrained by context length limitations, they show promise on short audio segments. This study establishes the first transcription-independent evaluation benchmark for audio segmentation and provides in-depth analysis of the interplay among transcription quality, acoustic properties, and model performance.

In meeting transcription, speech separation suffers from inter-channel leakage—particularly during single-active-speaker segments—and the impact of voice activity detection (VAD) and segmentation strategies on end-to-end performance remains poorly understood. This paper introduces a temporally sensitive leakage analysis framework, the first to explicitly model how separation quality depends on local temporal structure; it reveals that inter-channel leakage has limited impact on end-to-end transcription accuracy due to ASR’s inherent robustness. We systematically evaluate diverse segmentation strategies and, on LibriCSS, demonstrate that state-of-the-art speaker diarization reduces the gap to oracle segmentation by approximately one-third compared to energy-based VAD. Adopting a modular architecture—comprising VAD, speaker clustering, and separation—we train the ASR module exclusively on LibriSpeech and achieve SOTA word error rate on LibriCSS, validating strong cross-dataset generalization.

This work addresses the limitations of existing deepfake audio localization methods, which predominantly rely on boundary artifacts between genuine and forged segments and thus struggle to detect subtle, dispersed forgeries. To overcome this dependency on transitional regions, the paper proposes a Segment-Aware Learning (SAL) framework that enhances localization by modeling the internal structure of speech segments. SAL introduces segment-position labels to enable fine-grained frame-level supervision and incorporates a cross-segment mixing data augmentation strategy to encourage the model to focus on entire forged segments rather than solely on their boundaries. Experimental results demonstrate that SAL significantly outperforms current approaches across multiple datasets in both in-domain and out-of-domain settings, with particularly notable improvements in non-boundary regions.