What makes a good feedforward computational graph?

2025-02-10

AI Summary

This study investigates how feedforward computational graph structure affects neural network expressivity, addressing the problem that inappropriate graph topologies hinder learning of target functions. We propose two theoretically grounded, complementary metrics—fidelity and mixing time—and establish the first dual-metric analytical framework specifically for directed acyclic graphs (DAGs). Leveraging spectral graph theory, asymptotic analysis, and large-scale neural network experiments, we uncover interpretable relationships between graph structure and model performance: fundamental trade-offs exist between the two metrics across diverse DAGs, and both correlate strongly with empirical accuracy (mean Pearson correlation > 0.87). Our work fills a critical theoretical gap in graph rewiring research for DAG-structured networks and provides empirically validated, design-oriented principles for efficient graph architecture engineering.

Technology Category

• Machine Learning: Graph-based Machine Learning
• Reasoning under Uncertainty: Graphical Models
• Data Mining & Knowledge Management: Graph Mining, Social Network Analysis & Community

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Models for Web evolution
• Social Networks and Social Media: Computational social science

Abstract

As implied by the plethora of literature on graph rewiring, the choice of computational graph employed by a neural network can make a significant impact on its downstream performance. Certain effects related to the computational graph, such as under-reaching and over-squashing, may even render the model incapable of learning certain functions. Most of these effects have only been thoroughly studied in the domain of undirected graphs; however, recent years have seen a significant rise in interest in feedforward computational graphs: directed graphs without any back edges. In this paper, we study the desirable properties of a feedforward computational graph, discovering two important complementary measures: fidelity and mixing time, and evaluating a few popular choices of graphs through the lens of these measures. Our study is backed by both theoretical analyses of the metrics' asymptotic behaviour for various graphs, as well as correlating these metrics to the performance of trained neural network models using the corresponding graphs.

Problem

• Impact of computational graph on neural network performance
• Study of feedforward computational graph properties
• Evaluation of graph fidelity and mixing time measures

Innovation

• Feedforward computational graph analysis
• Fidelity and mixing time measures
• Theoretical and empirical correlation