Weight-Parameterization in Continuous Time Deep Neural Networks for Surrogate Modeling

2025-07-29

AI Summary

To address the challenge of jointly achieving high expressivity, numerical stability, and computational efficiency in time-varying weight modeling for continuous-time deep learning (e.g., neural ODEs), this paper proposes a low-dimensional parameterization method based on Legendre orthogonal polynomials. Specifically, time-varying weights are constrained to a subspace spanned by a finite number of Legendre basis functions. This formulation preserves strong representational capacity while significantly improving training stability and reducing memory and computational overhead. Integrated with the ODE-ResNet architecture, the approach supports both discrete-then-optimize and optimize-then-discrete training paradigms. Empirical evaluation on three high-dimensional physics benchmark tasks demonstrates that, compared to unconstrained models and monomial-basis parameterizations, our method achieves comparable or superior accuracy while accelerating training convergence and reducing average computational cost by 30–50%.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Computer Vision: Learning & Optimization for CV
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Continuous-time deep learning models, such as neural ordinary differential equations (ODEs), offer a promising framework for surrogate modeling of complex physical systems. A central challenge in training these models lies in learning expressive yet stable time-varying weights, particularly under computational constraints. This work investigates weight parameterization strategies that constrain the temporal evolution of weights to a low-dimensional subspace spanned by polynomial basis functions. We evaluate both monomial and Legendre polynomial bases within neural ODE and residual network (ResNet) architectures under discretize-then-optimize and optimize-then-discretize training paradigms. Experimental results across three high-dimensional benchmark problems show that Legendre parameterizations yield more stable training dynamics, reduce computational cost, and achieve accuracy comparable to or better than both monomial parameterizations and unconstrained weight models. These findings elucidate the role of basis choice in time-dependent weight parameterization and demonstrate that using orthogonal polynomial bases offers a favorable tradeoff between model expressivity and training efficiency.

Problem

• Learning stable time-varying weights in neural ODEs
• Reducing computational cost in surrogate modeling
• Improving training dynamics with orthogonal polynomial bases

Innovation

• Polynomial basis functions constrain weight evolution
• Legendre bases enhance training stability and efficiency
• Orthogonal bases balance expressivity and computational cost