Beyond the Golden Teacher: Enhancing Graph Learning through LLM-GNN Co-teaching

2026-06-09

AI Summary

This work addresses the challenges of few-shot learning on textual attributed graphs (TAGs), where graph neural networks (GNNs) struggle with cold-start nodes, large language models (LLMs) are susceptible to textual ambiguity, and existing approaches relying on a single “gold teacher” propagate blind spots. To overcome these limitations, the authors propose a bidirectional LLM-GNN collaborative teaching framework that abandons the fixed-teacher assumption and instead enables joint training through iterative cross-model pseudo-label exchange and dynamic preference optimization. A key innovation is round-level pseudo-label preference optimization (RPL-PO), which constructs DPO-style preference pairs by tracking prediction trajectories from inter-model disagreement to consensus. Evaluated on six benchmark datasets, the method achieves significant gains over state-of-the-art approaches, improving 3-shot accuracy by 7.86% on Cora and 7.73% on ogbn-arxiv, while also demonstrating strong generalization to 5-shot and zero-shot cross-dataset settings.

Abstract

Text-attributed graphs (TAGs) underlie real-world applications such as citation networks, social media, and e-commerce. Few-shot graph learning on TAGs is hard: with only a handful of labels per class and the rest of the graph unannotated, neither GNNs nor LLMs can learn well on their own. GNNs read topology and fail on cold nodes; LLMs read text and fail on text-ambiguous nodes. Existing LLM-GNN methods all follow the same recipe: designate one model as the golden teacher and use its outputs (e.g., features or pseudo-labels) to supervise the other. We argue this golden-teacher assumption breaks under sparse supervision: neither model is golden, and treating either as such transfers its blind spots into the student. We therefore ask: can we avoid designating either model as the golden teacher, and still perform effective graph learning? We answer with LLM-GNN Co-Teaching, a bidirectional co-teaching framework in which neither model is fixed as teacher. The GNN and LLM exchange their most confident pseudo-labels under an architecture-specific small-loss criterion, and both update every round. Supervision is then mined from the trajectory: whenever a node moves from cross-model contradiction at round t to cross-model agreement at round t+1, the LLM's two answers on the same input form a preference pair (old contradicting self < new peer-endorsed self) for DPO training. We call this Round-based Pseudo-Label Preference Optimization (RPL-PO). On six benchmarks, LLM-GNN Co-Teaching consistently outperforms GNN-as-Judge and all prior methods, with absolute 3-shot gains of 7.86% on Cora and 7.73% on ogbn-arxiv; improvements carry over to 5-shot and to zero-shot cross-dataset transfer. Error-structure analysis further shows that abandoning the golden-teacher assumption substantially improves the LLM's graph learning capability on challenging samples.

Problem

• few-shot graph learning
• text-attributed graphs
• LLM-GNN co-teaching
• sparse supervision
• pseudo-labeling

Innovation

• LLM-GNN Co-Teaching
• Text-attributed Graphs
• Few-shot Graph Learning
• Pseudo-label Preference Optimization
• Direct Preference Optimization (DPO)