DeltaFlow: Noise-Adaptive Bidirectional Gated Delta Networks for Embedded Language Flows

2026-08-02

AI Summary

This work addresses the high computational cost of Embedded Language Flow (ELF), which relies on fully non-causal attention for iterative denoising. The authors propose DeltaFlow—a novel noise-adaptive, bidirectional gated Delta network architecture that introduces gated Delta mechanisms into the ELF framework for the first time, enabling efficient context modeling through alternating or parallel bidirectional scans. Key innovations include noise-adaptive memory control and a Temporal State Consistency (TSC) scheduling mechanism, integrated with an SDE sampler. Evaluated on OpenWebText, DeltaFlow-P reduces generation perplexity from 24.218 to 21.228 while preserving comparable unigram entropy, achieves a 2.72× throughput speedup at 16k sequence length, and requires only 36 billion training tokens.

Abstract

Embedded Language Flows (ELF) rely primarily on full non-causal attention for iterative denoising, repeatedly incurring quadratic sequence-mixing cost at each sampling step. Gated Delta Networks (GDNs) provide an efficient recurrent alternative, but their standard causal formulation cannot directly capture the bidirectional context required by ELF. We introduce DeltaFlow, a noise-adaptive bidirectional GDN backbone for continuous language denoising. We study two variants: DeltaFlow-A, which alternates scan directions across layers, and DeltaFlow-P, which performs parallel forward and backward scans within each layer. We further introduce noise-adaptive memory control and scheduled Temporal State Consistency (TSC) to stabilize hidden representations across nearby noise levels. On OpenWebText, using a 32-step stochastic differential equation sampler, DeltaFlow-P reduces generated perplexity from 24.218 for the full-attention ELF baseline to 21.228 while maintaining comparable unigram entropy, with 36B training-token exposure compared with 45B for the baseline. In a denoiser-only benchmark, DeltaFlow-P achieves a 2.72x throughput speedup over the full-attention baseline at a sequence length of 16k. These results show that DeltaFlow is a promising alternative to dense attention for efficient continuous language denoising.

Problem

• Embedded Language Flows
• non-causal attention
• quadratic complexity
• bidirectional context
• language denoising

Innovation

• DeltaFlow
• Gated Delta Networks
• bidirectional recurrent architecture
• noise-adaptive memory control
• Temporal State Consistency