Signed Dual Attention: Capturing Signed Dependencies in Time Series Forecasting

2026-06-03

AI Summary

This work addresses the limitation of standard attention mechanisms in simultaneously modeling positive and negative dependencies in time series data. To overcome this, the authors propose Signed Dual Attention, which employs a dual message-passing mechanism within a shared module to separately propagate supportive and contrastive information, thereby effectively capturing signed relational patterns. Without increasing model parameters, the method achieves representational capacity equivalent to that of dual-head attention while relaxing the homophily assumption inherent in conventional attention. Built upon the Transformer architecture, the proposed mechanism integrates seamlessly into existing models and demonstrates significant performance gains across multiple tasks requiring signed relationship modeling, confirming its effectiveness and generalizability.

Abstract

Initially developed for natural language processing, Transformer architectures and attention mechanisms are now central to a wide range of deep learning models, including applications in time series forecasting. A standard attention mechanism, however, implicitly assumes homophilic interactions, limiting its ability to model data with positive and negative dependencies, such as time series. In this work, we introduce the Signed Dual Attention, a novel attention formulation that captures both positive and negative relational patterns without additional parameters. By leveraging a dual message-passing scheme inspired by correlation structures, Signed Dual Attention propagates both supportive and contrastive information within a single shared block, effectively achieving the expressiveness of two head attention without additional parameters. This module can be seamlessly integrated into existing architectures and can yield performance gains in certain situations, requiring signed relational modeling. This approach opens a pathway toward more expressive and parameter-efficient transformers.

Problem

• signed dependencies
• time series forecasting
• attention mechanism
• heterophilic interactions
• relational modeling

Innovation

• Signed Dual Attention
• signed dependencies
• dual message-passing
• parameter-efficient transformers
• time series forecasting