DINT Transformer

2025-01-29

AI Summary

Existing DIFF Transformers struggle to model global dependencies and suffer from numerical instability in attention computations. To address these issues, we propose a novel differential-integral attention mechanism: differential operations enhance local robustness, while integral-based importance scoring explicitly captures global contextual information; moreover, we introduce parametric row-wise normalization—the first such approach—to explicitly ensure numerical stability of the attention matrix. This mechanism requires no additional positional encodings or long-range extension modules and is natively compatible with standard Transformer architectures. Empirical evaluations on long-context language modeling and critical information retrieval tasks demonstrate substantial improvements in accuracy (+2.1%–3.8%) and noise robustness. These results validate the proposed mechanism as a general-purpose, numerically stable, and computationally efficient next-generation attention paradigm.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Safety and Robustness
• Machine Learning: Mixture of Experts (MoE)

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

DIFF Transformer addresses the issue of irrelevant context interference by introducing a differential attention mechanism that enhances the robustness of local attention. However, it has two critical limitations: the lack of global context modeling, which is essential for identifying globally significant tokens, and numerical instability due to the absence of strict row normalization in the attention matrix. To overcome these challenges, we propose DINT Transformer, which extends DIFF Transformer by incorporating a differential-integral mechanism. By computing global importance scores and integrating them into the attention matrix, DINT Transformer improves its ability to capture global dependencies. Moreover, the unified parameter design enforces row-normalized attention matrices, improving numerical stability. Experimental results demonstrate that DINT Transformer excels in accuracy and robustness across various practical applications, such as long-context language modeling and key information retrieval. These results position DINT Transformer as a highly effective and promising architecture.

Problem

• Global Information Capturing
• Attention Distribution
• Computational Stability

Innovation

• Differential Integral
• global context
• computational stability