InfGraND: An Influence-Guided GNN-to-MLP Knowledge Distillation

2026-01-12

Trans. Mach. Learn. Res.

AI Summary

This work addresses the limited expressiveness of traditional multilayer perceptrons (MLPs) in low-latency or resource-constrained graph data scenarios, where existing GNN-to-MLP knowledge distillation approaches often overlook the importance of node structural roles. To bridge this gap, the authors propose a novel distillation framework that introduces graph structural influence as distillation weights, prioritizing knowledge transfer from structurally critical nodes. Furthermore, by precomputing multi-hop neighborhood features in a single pass and embedding them into the MLP input, the method enhances the student model’s structural awareness without incurring additional inference overhead. Evaluated across seven homophilic graph benchmarks under both transductive and inductive settings, the approach consistently outperforms current GNN-to-MLP distillation methods, demonstrating its suitability for real-world low-latency applications.

Technology Category

• Machine Learning: Graph-based Machine Learning
• Data Mining & Knowledge Management: Graph Mining, Social Network Analysis & Community
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Content-based information diffusion

Abstract

Graph Neural Networks (GNNs) are the go-to model for graph data analysis. However, GNNs rely on two key operations - aggregation and update, which can pose challenges for low-latency inference tasks or resource-constrained scenarios. Simple Multi-Layer Perceptrons (MLPs) offer a computationally efficient alternative. Yet, training an MLP in a supervised setting often leads to suboptimal performance. Knowledge Distillation (KD) from a GNN teacher to an MLP student has emerged to bridge this gap. However, most KD methods either transfer knowledge uniformly across all nodes or rely on graph-agnostic indicators such as prediction uncertainty. We argue this overlooks a more fundamental, graph-centric inquiry:"How important is a node to the structure of the graph?"We introduce a framework, InfGraND, an Influence-guided Graph KNowledge Distillation from GNN to MLP that addresses this by identifying and prioritizing structurally influential nodes to guide the distillation process, ensuring that the MLP learns from the most critical parts of the graph. Additionally, InfGraND embeds structural awareness in MLPs through one-time multi-hop neighborhood feature pre-computation, which enriches the student MLP's input and thus avoids inference-time overhead. Our rigorous evaluation in transductive and inductive settings across seven homophilic graph benchmark datasets shows InfGraND consistently outperforms prior GNN to MLP KD methods, demonstrating its practicality for numerous latency-critical applications in real-world settings.

Problem

• Knowledge Distillation
• Graph Neural Networks
• Multi-Layer Perceptrons
• Node Influence
• Graph Structure

Innovation

• Influence-guided
• Graph Neural Networks
• Knowledge Distillation
• Multi-Layer Perceptron
• Structural Awareness