Automatically Differentiable Nonlinear Tensor Networks (ADNTNs) for Exponential Compression of Deep Neural Networks

2026-05-28

AI Summary

This work addresses the challenge of deploying deep neural networks with massive parameter counts by proposing a structured weight generation method based on differentiable nonlinear tensor networks. Leveraging tree tensor networks (TTN), augmented TTN (aTTN), and multiscale entanglement renormalization ansatz (MERA), the approach integrates nonlinear activations and task-aware objectives to end-to-end train compact core tensors that implicitly generate full-weight matrices. The architecture enables hardware-aware tensor contraction scheduling, achieving per-layer compression ratios ranging from 2,000× to 77,000× on AlexNet and VGG-16 while matching or even surpassing the accuracy of the original dense models, thereby substantially improving model compressibility and deployment efficiency.

Abstract

We study Automatically Differentiable Nonlinear Tensor Networks (ADNTNs), a family of structured weight generators whose compact core tensors are trained end-to-end by reverse-mode automatic differentiation (AD). The approach can be viewed as a natural extension of low-rank adaptation and tensor factorisation: instead of using one low-rank matrix update, an ADNTN builds a large weight tensor through a hierarchy of small cores, nonlinear activations, and optional lateral mixing tensors. The paper focuses on three architectures: Tree Tensor Networks (TTNs), augmented TTNs (aTTNs) with boundary disentanglers, and Multi-scale Entanglement Renormalisation Ansatze (MERA). The formulation supports nonlinear activations, task-aware objectives, batching, and hardware-aware execution schedules. At the same time, the paper keeps a clear distinction between \emph{differentiating} a contraction program and making contraction free: AD does not remove the cost of large intermediates, poor contraction orders, or exact contraction of general loopy tensor networks. Extensive simulations on AlexNet and VGG-16 layers show per-layer compression ratios from roughly $2000\times$ to $77000\times$ in the studied settings, with accuracy often matching the dense baseline and, in several VGG-16 cases, improving it. These results are encouraging rather than final: they suggest that ADNTNs are a promising, mathematically structured, and hardware-aware route toward much smaller neural networks, provided that optimisation, contraction schedules, and deployment kernels are designed together.

Problem

• neural network compression
• tensor networks
• parameter efficiency
• model compression
• deep learning

Innovation

• Automatically Differentiable Tensor Networks
• Nonlinear Tensor Networks
• Exponential Model Compression
• Hardware-aware Execution
• End-to-end Training