Interplay Between Belief Propagation and Transformer: Differential-Attention Message Passing Transformer

2025-09-19

AI Summary

Neural decoding of LDPC codes faces challenges in modeling long-range dependencies over graph structures and effectively incorporating the principles of classical belief propagation (BP). Method: This paper proposes a novel Transformer-based neural decoder. It introduces a differentiable syndrome loss to enforce parity-check constraints end-to-end, and a differential attention mechanism that explicitly models bidirectional message passing between bit and check nodes—embedding BP’s message update rules into self-attention computations. Additionally, graph-structure-aware node embeddings and hierarchical supervision signals enable joint learning of the LDPC code’s global topology. Results: Experiments demonstrate that the proposed decoder significantly outperforms conventional BP and state-of-the-art neural decoders on short-to-medium-length LDPC codes, achieving 0.5–1.2 dB coding gain in bit error rate under AWGN channels.

Technology Category

Cognitive Modeling & Cognitive Systems: Neural Spike CodingMachine Learning: Deep Generative Models & AutoencodersNatural Language Processing: Learning & Optimization for NLP

Application Category

Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphsEconomics, Online Markets and Human Computation: Cost models of using LLMs in production systemsSearch and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Transformer-based neural decoders have emerged as a promising approach to error correction coding, combining data-driven adaptability with efficient modeling of long-range dependencies. This paper presents a novel decoder architecture that integrates classical belief propagation principles with transformer designs. We introduce a differentiable syndrome loss function leveraging global codebook structure and a differential-attention mechanism optimizing bit and syndrome embedding interactions. Experimental results demonstrate consistent performance improvements over existing transformer-based decoders, with our approach surpassing traditional belief propagation decoders for short-to-medium length LDPC codes.

Problem

• Integrates belief propagation with transformer for decoding
• Optimizes bit and syndrome embedding interactions
• Improves performance for LDPC error correction codes

Innovation

• Integrates belief propagation with transformer
• Uses differential-attention mechanism optimization
• Leverages differentiable syndrome loss function