Memory-Computation Tradeoffs in Semi Amortized Parametric Optimization

2026-07-22

AI Summary

This work investigates the trade-off between memory and computation under a fixed online computational budget, specifically quantifying the amount of offline storage required to achieve a target accuracy. The authors propose a semi-amortized parameter optimization framework that stores a limited set of pre-solved instances offline and, at test time, retrieves the most relevant one to warm-start K steps of projected gradient descent. For smooth convex problems, they establish matching upper and lower bounds on memory complexity in the strongly convex setting, revealing a phase transition phenomenon: under β-growth conditions, the benefit of additional memory vanishes once K exceeds a certain threshold. A general framework is provided to quantify the memory cost necessary for acceleration. The theoretical findings are validated through parametric ridge regression experiments, accurately characterizing the interplay among memory, computation, and solution accuracy.

Abstract

Learning-enabled decision systems often use offline data or computation to reduce online compute cost. Despite the empirical success of such approaches, there is limited general understanding of how much offline information is needed to achieve a desired accuracy under a fixed online computation budget. We study this question through the lens of amortized parametric optimization: an offline phase stores a finite memory of solved problem instances, and an online phase produces a solution to a new instance by retrieving a warm start and applying $K$ steps of projected gradient descent. We analyze this setup for smooth convex parametric optimization over a compact domain, using a nonparametric predictor built from the stored offline solutions. For $μ$-strongly convex objectives, we establish matching upper and lower bounds on the memory required to guarantee $\varepsilon$-accuracy under a fixed online iteration budget $K$. For convex objectives satisfying a $β$-growth condition ($β>2$), we obtain near-matching bounds and identify a phase transition in $K$ beyond which additional memory provides no benefit. We further provide a general proof framework that (i) explicitly quantifies the memory cost of acceleration---how much offline memory is required to achieve a prescribed speedup over the unaided online optimizer---and (ii) identifies two key quantities driving this cost: the convergence rate of the online optimizer and the Lipschitz sensitivity of the solution map to the problem parameter. Experiments on parameterized ridge regression confirm the predicted memory--computation--accuracy tradeoffs.

Problem

• memory-computation tradeoff
• amortized optimization
• parametric optimization
• offline-online learning
• warm-start optimization

Innovation

• amortized optimization
• memory-computation tradeoff
• parametric optimization
• offline-online learning
• warm-start acceleration