🤖 AI Summary
Repeatedly solving similar optimization problems incurs substantial computational overhead.
Method: This paper proposes amortized optimization—a learning-based paradigm that predicts approximate solutions for new problem instances by leveraging structural information from historical problems, thereby accelerating optimization. We establish the first unified theoretical framework for amortized optimization, encompassing variational inference, meta-learning, and optimal transport. Our approach integrates deep neural networks, variational inference, gradient-based meta-learning, and convex optimization modeling to enable end-to-end differentiable optimizer design.
Contribution/Results: The framework unifies diverse inference and learning tasks under a coherent theoretical lens; it introduces a general design paradigm for differentiable optimizers; and empirical evaluation demonstrates speedups of several orders of magnitude over conventional optimizers in variational inference, reinforcement learning, and sparse coding—while maintaining strong generalization across tasks.
📝 Abstract
Optimization is a ubiquitous modeling tool and is often deployed in settings which repeatedly solve similar instances of the same problem. Amortized optimization methods use learning to predict the solutions to problems in these settings, exploiting the shared structure between similar problem instances. These methods have been crucial in variational inference and reinforcement learning and are capable of solving optimization problems many orders of magnitudes times faster than traditional optimization methods that do not use amortization. This tutorial presents an introduction to the amortized optimization foundations behind these advancements and overviews their applications in variational inference, sparse coding, gradient-based meta-learning, control, reinforcement learning, convex optimization, optimal transport, and deep equilibrium networks. The source code for this tutorial is available at https://github.com/facebookresearch/amortized-optimization-tutorial.
