Efficient Finite Initialization for Tensorized Neural Networks

2023-09-11

arXiv.org

AI Summary

In tensorized neural networks, initializing high-dimensional tensor layers often leads to parameter explosion or vanishing norms, while explicit parameter storage is infeasible due to memory constraints. Method: This paper proposes a norm-constrained finite initialization method that innovatively combines the Frobenius norm with partial linear element-wise norms, enabling iterative tensor normalization with intermediate computation reuse—applicable to fully connected, large-scale, and implicitly parameterized tensor layers. Contribution/Results: The method significantly mitigates gradient explosion and enhances training stability across diverse tensorized architectures. Its open-source Python implementation has been integrated into the i3BQuantum library—a quantum-inspired variational algorithm framework—demonstrating strong reusability, extensibility, and practical utility for scalable tensorized deep learning.

Abstract

We present a novel method for initializing layers of tensorized neural networks in a way that avoids the explosion of the parameters of the matrix it emulates. The method is intended for layers with a high number of nodes in which there is a connection to the input or output of all or most of the nodes, we cannot or do not want to store/calculate all the elements of the represented layer and they follow a smooth distribution. This method is equally applicable to normalize general tensor networks in which we want to avoid overflows. The core of this method is the use of the Frobenius norm and the partial lineal entrywise norm of reduced forms of the layer in an iterative partial form, so that it has to be finite and within a certain range. These norms are efficient to compute, fully or partially for most cases of interest. In addition, the method benefits from the reuse of intermediate calculations. We apply the method to different layers and check its performance. We create a Python function to run it on an arbitrary layer, available in a Jupyter Notebook in the i3BQuantum repository: https://github.com/i3BQuantumTeam/Q4Real/blob/e07c827651ef16bcf74590ab965ea3985143f891/Quantum-Inspired%20Variational%20Methods/TN_Normalizer.ipynb

Problem

• Initialize tensorized neural networks efficiently
• Prevent divergence or zero norm in subnetworks
• Optimize scaling for MPS/TT and MPO/TT-M layers

Innovation

• Uses partial Frobenius and lineal norms
• Iteratively normalizes subnetworks' norms
• Optimizes MPS/TT and MPO/TT-M layers