SVDformer: Direction-Aware Spectral Graph Embedding Learning via SVD and Transformer

2025-08-18

AI Summary

Existing directed graph neural networks (GNNs) are limited by isotropic neighborhood aggregation and fixed local spectral filters, hindering joint modeling of directional semantics and global graph structure. To address this, we propose SVDformer—a novel architecture that deeply integrates singular value decomposition (SVD) with the Transformer. Specifically, it employs left and right singular vectors as direction-aware projection bases and singular values as learnable scaling factors to explicitly capture multi-scale interactions between in-edges and out-edges. Crucially, SVDformer abandons predefined spectral kernel functions and instead enables end-to-end learning of low-pass and high-pass graph filters. Evaluated on node classification across six directed graph benchmarks, SVDformer consistently outperforms state-of-the-art GNNs and direction-aware methods. These results demonstrate its effectiveness and superiority in learning direction-sensitive representations and modeling global structural dependencies.

Technology Category

• Machine Learning: Graph-based Machine Learning
• Data Mining & Knowledge Management: Graph Mining, Social Network Analysis & Community
• Computer Vision: Representation Learning for Vision

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding

Abstract

Directed graphs are widely used to model asymmetric relationships in real-world systems. However, existing directed graph neural networks often struggle to jointly capture directional semantics and global structural patterns due to their isotropic aggregation mechanisms and localized filtering mechanisms. To address this limitation, this paper proposes SVDformer, a novel framework that synergizes SVD and Transformer architecture for direction-aware graph representation learning. SVDformer first refines singular value embeddings through multi-head self-attention, adaptively enhancing critical spectral components while suppressing high-frequency noise. This enables learnable low-pass/high-pass graph filtering without requiring spectral kernels. Furthermore, by treating singular vectors as directional projection bases and singular values as scaling factors, SVDformer uses the Transformer to model multi-scale interactions between incoming/outgoing edge patterns through attention weights, thereby explicitly preserving edge directionality during feature propagation. Extensive experiments on six directed graph benchmarks demonstrate that SVDformer consistently outperforms state-of-the-art GNNs and direction-aware baselines on node classification tasks, establishing a new paradigm for learning representations on directed graphs.

Problem

• Addresses isotropic aggregation in directed graph neural networks
• Captures directional semantics and global structural patterns
• Enables learnable graph filtering without spectral kernels

Innovation

• SVDformer combines SVD and Transformer architecture
• Uses multi-head attention to refine spectral embeddings
• Models directional interactions through attention weights