Converting MLPs into Polynomials in Closed Form

2025-02-03

AI Summary

Deep neural networks (DNNs) suffer from a fundamental interpretability bottleneck due to their black-box nature. Method: This work establishes a rigorous polynomial modeling framework for multilayer perceptrons (MLPs) with gated linear units (GLUs), deriving closed-form least-squares optimal polynomial approximations of arbitrary order. It introduces a linear-algebraic interpretability paradigm integrating polynomial expansion, eigenvalue decomposition (EVD)/singular value decomposition (SVD), and dynamic complexity modeling to characterize the progressive evolution—from low-order (linear/quadratic) to high-order representations—during training. Contributions/Results: Experiments show quadratic approximations explain over 95% of output variance (R² > 0.95) at convergence; SVD-driven adversarial example generation is theoretically traceable; and feature visualizations significantly enhance human interpretability. This is the first polynomial-based DNN interpretability approach offering theoretical closed-form solutions, quantitatively verifiable metrics, and structurally traceable analysis.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Computer Vision: Interpretability, Explainability, and Transparency
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Recent work has shown that purely quadratic functions can replace MLPs in transformers with no significant loss in performance, while enabling new methods of interpretability based on linear algebra. In this work, we theoretically derive closed-form least-squares optimal approximations of feedforward networks (multilayer perceptrons and gated linear units) using polynomial functions of arbitrary degree. When the $R^2$ is high, this allows us to interpret MLPs and GLUs by visualizing the eigendecomposition of the coefficients of their linear and quadratic approximants. We also show that these approximants can be used to create SVD-based adversarial examples. By tracing the $R^2$ of linear and quadratic approximants across training time, we find new evidence that networks start out simple, and get progressively more complex. Even at the end of training, however, our quadratic approximants explain over 95% of the variance in network outputs.

Problem

• Neural Networks
• Polynomial Representation
• Interpretability

Innovation

• Polynomial Approximation
• Feature Decomposition
• Complexity Analysis