🤖 AI Summary
This work addresses rank collapse in message-passing neural networks (MPNNs), a generalization of the over-smoothing phenomenon, by identifying two underlying mechanisms: shared component amplification (SCA) and component dominance (CD). For the first time, the paper decomposes over-smoothing into these distinct factors and establishes a theoretical connection between MPNNs and personalized PageRank. To mitigate rank collapse without altering the backbone architecture, the authors propose a multi-relational splitting (MRS) framework and introduce multi-input multi-output graph convolution (MIMO-GC) along with its local variant, LMGC, which inject multi-relational structure into standard MPNNs. Furthermore, they design an infinitely deep MPNN variant grounded in personalized PageRank. The proposed methods effectively alleviate rank collapse, enable infinite-depth propagation, preserve initial features, and significantly enhance model expressiveness and stability.
📝 Abstract
While message-passing neural networks (MPNNs) have shown promising results, their real-world impact remains limited. Although various limitations have been identified, their theoretical foundations remain poorly understood, leading to fragmented research efforts. In this thesis, we provide an in-depth theoretical analysis and identify several key properties limiting the performance of MPNNs. Building on these findings, we propose several frameworks that address these shortcomings. We identify two properties exhibited by many MPNNs: shared component amplification (SCA), where each message-passing iteration amplifies the same components across all feature channels, and component dominance (CD), where a single component gets increasingly amplified as more message-passing steps are applied. These properties lead to the observable phenomenon of rank collapse of node representations, which generalizes the established over-smoothing phenomenon. By generalizing and decomposing over-smoothing, we enable a deeper understanding of MPNNs, more targeted solutions, and more precise communication within the field. To avoid SCA, we show that utilizing multiple computational graphs or edge relations is necessary. Our multi-relational split (MRS) framework transforms any existing MPNN into one that leverages multiple edge relations. Additionally, we introduce the spectral graph convolution for multiple feature channels (MIMO-GC), which naturally uses multiple computational graphs. A localized variant, LMGC, approximates the MIMO-GC while inheriting its beneficial properties. To address CD, we demonstrate a close connection between MPNNs and the PageRank algorithm. Based on personalized PageRank, we propose a variant of MPNNs that allows for infinitely many message-passing iterations, while preserving initial node features. Collectively, these results deepen the theoretical understanding of MPNNs.