WOMBAT: Whitebox Oracle for Molecular Benchmarking and Attribution Testing

2026-09-30

AI Summary

This study addresses the challenge that existing GNN explainers struggle to distinguish their own failures from model shortcut learning, leaving attribution evaluation without reliable ground truth. To overcome this, it proposes the first white-box GNN benchmark framework based on handcrafted weights, constructing fourteen benchmark datasets with known attribution ground truths. By integrating SMARTS motifs, message-passing networks, and various post-hoc explanation methods, the framework enables verifiable testing of explainer errors. All datasets and code are open-sourced. Furthermore, this work reveals an attribution diffusion phenomenon exhibited by Integrated Gradients (IG) on specific models. Overall, these contributions provide a rigorous benchmarking tool and novel insights for evaluating GNN interpretability.

Abstract

When a graph neural network (GNN) explainer produces an unexpected attribution on a molecule, the attribution alone cannot reveal whether the explainer has failed or the model has learned a shortcut. We introduce WOMBAT, a benchmark of 14 whitebox GNNs, each with message-passing weights set by hand to detect a specific SMARTS motif. Each model's decision rule is known by construction, providing attribution ground truth against which explainer errors can be identified and studied. We validate the models on millions of PubChem molecules and evaluate post-hoc explainers including GNNExplainer, PGExplainer, and Integrated Gradients. Guided by our qualitative analysis, we construct a model that causes Integrated Gradients to spread attribution across the graph, even though the model reliably detects the intended motif. We release the dataset, models, and evaluation code to help researchers in the development of newer XAI tools for GNNs.

Problem

• Graph Neural Networks
• Explainability
• Attribution Ground Truth
• Molecular Benchmarking
• Shortcut Learning

Innovation

• Whitebox GNN
• Molecular Benchmarking
• Attribution Ground Truth
• Explainable AI
• SMARTS Motif