🤖 AI Summary
This study addresses the problem of reducing the oracle complexity of gradient descent by leveraging the smoothness of the empirical risk minimization loss function over the data domain. To this end, the authors propose an inexact gradient method, termed PPI-GD, which approximates the full gradient via piecewise tensor-product polynomial interpolation constructed from isometric samples in the data domain, thereby significantly decreasing the number of gradient queries. This work is the first to integrate data-domain smoothness with interpolation techniques for optimizing oracle complexity and extends bicubic spline error analysis to the multivariate setting. Theoretical results demonstrate that, when the loss function is sufficiently smooth and the data dimension scales polylogarithmically with the sample size, PPI-GD outperforms existing gradient descent variants in both strongly convex and non-convex regimes.
📝 Abstract
Recent work on first-order optimizers for empirical risk minimization (ERM) has suggested that smoothness of ERM loss functions in the training data, rather than in the optimization parameters, can be leveraged to improve the oracle complexity of gradient descent (GD) methods. In this paper, we propose an inexact gradient method, piecewise polynomial interpolation-based gradient descent (PPI-GD), which approximates the full gradient in each iteration by querying the first-order oracle at equidistant points in the data domain to construct polynomial interpolants of the resulting gradient samples over appropriately sized patches of the data domain. We analyze the oracle complexity of PPI-GD for strongly convex and non-convex loss functions when the data space dimension is bounded by a polylogarithmic function of the number of training samples, and find it to outperform several GD variants in key regimes when the loss function is sufficiently smooth. Furthermore, our analysis extends several techniques from the error analysis of bicubic spline interpolants to the setting of $d$-variate tensor product polynomial interpolants which may be of independent interest in interpolation analysis.