A Weisfeiler-Leman Characterization of Global-Attention Graph Transformers for Mixed-Integer Linear Programs

2026-07-20

AI Summary

This work investigates the expressive limits of global attention-based graph foundation models in representing mixed-integer linear programming (MILP) problems, with a focus on their ability to distinguish between non-isomorphic graph instances that are 1-WL equivalent. By integrating the 1-dimensional Weisfeiler–Leman (1-WL) graph isomorphism test, analysis of symmetric multiset functions, and various graph encoders—including Graphormer and GraphGPS—augmented with random walk positional encodings, the study reveals for the first time that prevailing global attention architectures are universally constrained by 1-WL equivalence when encoding MILP structures. The authors introduce an encoder-agnostic diagnostic framework and validate it across ten diverse architectures, demonstrating that all produce identical embeddings for 1-WL-equivalent non-isomorphic graphs, while the incorporation of positional information partially alleviates this limitation.

Abstract

Graph foundation models (GFMs) with global attention are increasingly used to represent mixed-integer linear programs (MILPs), aiming to capture structure beyond the locality of standard graph neural networks. We study their expressive power through graph isomorphism testing, asking which MILP instances they map to identical representations. We prove that a broad class of hierarchical graph transformers combining global linear attention, edge-weighted cross-attention, and bipartite message passing is bounded by the one-dimensional Weisfeiler-Leman (1-WL) test: under any parameter setting, 1-WL-equivalent MILP graphs receive identical graph embeddings. Our compositional proof shows that each architectural component is a symmetric multiset function and thus preserves 1-WL equivalence. We validate this characterization across ten diverse graph encoders, including Graphormer-, GraphGPS-, Set-Transformer-, and Gasse-style models. Across model capacities, graph scales, and pooling operators, every tested encoder maps 1-WL-equivalent non-isomorphic graph pairs to numerically identical embeddings. Consequently, graph invariants that vary within a 1-WL equivalence class cannot be recovered from these representations. We further show that expressiveness beyond 1-WL arises from input encoding rather than attention: random-walk positional encodings separate the constructed pairs, while additional constructions expose the limits of this remedy. These results characterize the expressive power of global-attention GFMs and provide an encoder-agnostic diagnostic for detecting 1-WL-induced representation equivalence.

Problem

• graph foundation models
• mixed-integer linear programs
• Weisfeiler-Leman test
• expressive power
• graph isomorphism

Innovation

• Weisfeiler-Leman
• graph transformers
• expressive power
• global attention
• MILP representation