Balancing Graph Embedding Smoothness in Self-Supervised Learning via Information-Theoretic Decomposition

2025-04-16

AI Summary

Existing graph self-supervised learning (SSL) methods exhibit polarized embedding smoothness—either over-smoothing or over-sharpening—leading to unstable downstream performance. To address this, we first formulate the graph SSL objective through an information-theoretic lens, decomposing it into three interpretable, mutually complementary losses: neighbor consistency loss, redundancy minimization loss, and distribution divergence loss—thereby revealing the theoretical root cause of smoothness imbalance. Building upon this decomposition, we propose a tunable multi-objective collaborative optimization framework that dynamically balances smoothness across structural and semantic dimensions. Our method achieves state-of-the-art performance on node classification and link prediction across multiple real-world graph benchmarks, consistently outperforming leading graph SSL approaches. It offers both strong theoretical interpretability—grounded in information theory—and cross-task robustness, demonstrating stable generalization without task-specific architectural modifications.

Technology Category

• Machine Learning: Graph-based Machine Learning
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Self-supervised learning (SSL) in graphs has garnered significant attention, particularly in employing Graph Neural Networks (GNNs) with pretext tasks initially designed for other domains, such as contrastive learning and feature reconstruction. However, it remains uncertain whether these methods effectively reflect essential graph properties, precisely representation similarity with its neighbors. We observe that existing methods position opposite ends of a spectrum driven by the graph embedding smoothness, with each end corresponding to outperformance on specific downstream tasks. Decomposing the SSL objective into three terms via an information-theoretic framework with a neighbor representation variable reveals that this polarization stems from an imbalance among the terms, which existing methods may not effectively maintain. Further insights suggest that balancing between the extremes can lead to improved performance across a wider range of downstream tasks. A framework, BSG (Balancing Smoothness in Graph SSL), introduces novel loss functions designed to supplement the representation quality in graph-based SSL by balancing the derived three terms: neighbor loss, minimal loss, and divergence loss. We present a theoretical analysis of the effects of these loss functions, highlighting their significance from both the SSL and graph smoothness perspectives. Extensive experiments on multiple real-world datasets across node classification and link prediction consistently demonstrate that BSG achieves state-of-the-art performance, outperforming existing methods. Our implementation code is available at https://github.com/steve30572/BSG.

Problem

• Balancing graph embedding smoothness in self-supervised learning
• Decomposing SSL objectives via information-theoretic framework
• Improving performance across diverse downstream graph tasks

Innovation

• Balances graph embedding smoothness via information-theoretic decomposition
• Introduces novel loss functions: neighbor, minimal, divergence
• Achieves state-of-the-art performance in node and link tasks