🤖 AI Summary
Approximating high-dimensional, low-smoothness functions—common in cross-domain learning (e.g., invariant learning, transfer learning, SAR imaging)—remains challenging due to limitations of conventional symmetric or positive-definite kernels.
Method: This paper proposes a neural network framework based on asymmetric, irregular kernels, breaking away from traditional kernel symmetry/positivity constraints. It systematically constructs generalized translation networks and rotationally banded function kernels—novel asymmetric kernel architectures—and establishes their approximation theory. It further introduces the ReLU<sup>r</sup> activation (with non-integer r > 0) and derives its uniform approximation error bound for Sobolev functions.
Contribution/Results: Leveraging asymmetric kernel decomposition and Sobolev space analysis, the framework yields tight approximation error estimates for low-smooth, high-dimensional functions. Empirically, it achieves significantly improved cross-domain generalization accuracy under small-sample and low-regularity conditions.
📝 Abstract
For the past 30 years or so, machine learning has stimulated a great deal of research in the study of approximation capabilities (expressive power) of a multitude of processes, such as approximation by shallow or deep neural networks, radial basis function networks, and a variety of kernel based methods. Motivated by applications such as invariant learning, transfer learning, and synthetic aperture radar imaging, we initiate in this paper a general approach to study the approximation capabilities of kernel based networks using non-symmetric kernels. While singular value decomposition is a natural instinct to study such kernels, we consider a more general approach to include the use of a family of kernels, such as generalized translation networks (which include neural networks and translation invariant kernels as special cases) and rotated zonal function kernels. Naturally, unlike traditional kernel based approximation, we cannot require the kernels to be positive definite. In particular, we obtain estimates on the accuracy of uniform approximation of functions in a Sobolev class by ReLU$^r$ networks when $r$ is not necessarily an integer. Our general results apply to the approximation of functions with small smoothness compared to the dimension of the input space.