Enhanced Soups for Graph Neural Networks

2025-03-14

AI Summary

Existing graph neural network (GNN) ensemble methods suffer from slow inference, high memory overhead, and poor scalability. To address these limitations, this paper proposes a gradient-descent-based learnable Soup strategy—introducing *Learned Souping* and its block-wise variant, *Partition Learned Souping*. Unlike conventional fixed-weight ensembling, our approach jointly optimizes model parameters and weighting coefficients in an end-to-end manner, achieving superior accuracy with significantly reduced computational and memory costs. On multiple OGB benchmarks, the method improves test accuracy by up to 1.2%, accelerates inference by 2.1× overall, and on ogbn-products, the Partition variant achieves a 24.5× speedup and 76% memory reduction—overcoming the efficiency bottlenecks inherent in standard Soup methods.

Technology Category

• Machine Learning: Ensemble Methods
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Graph Neural Networks (GNN) have demonstrated state-of-the-art performance in numerous scientific and high-performance computing (HPC) applications. Recent work suggests that"souping"(combining) individually trained GNNs into a single model can improve performance without increasing compute and memory costs during inference. However, existing souping algorithms are often slow and memory-intensive, which limits their scalability. We introduce Learned Souping for GNNs, a gradient-descent-based souping strategy that substantially reduces time and memory overhead compared to existing methods. Our approach is evaluated across multiple Open Graph Benchmark (OGB) datasets and GNN architectures, achieving up to 1.2% accuracy improvement and 2.1X speedup. Additionally, we propose Partition Learned Souping, a novel partition-based variant of learned souping that significantly reduces memory usage. On the ogbn-products dataset with GraphSAGE, partition learned souping achieves a 24.5X speedup and a 76% memory reduction without compromising accuracy.

Problem

• Improve GNN performance via efficient model combination
• Reduce time and memory overhead in souping algorithms
• Enhance scalability of GNNs without compromising accuracy

Innovation

• Gradient-descent-based souping strategy for GNNs
• Partition-based variant reduces memory usage significantly
• Achieves speedup and accuracy improvement on benchmarks