🤖 AI Summary
Deep learning model scaling has led to surging computational and energy demands, severely hindering deployment in resource-constrained environments. To address this, we propose a systematic approximation framework for efficient and scalable deep learning. Our method reformulates discrete constraints—such as pruning and quantization—as differentiable continuous relaxations, enabling fine-grained joint compression; introduces a lightweight neural architecture search algorithm based on parameter sharing and implicit recurrent structures; and enhances adaptive optimizers to accelerate hyperparameter convergence. The framework supports end-to-end joint training. Evaluated on image classification, language modeling, and generative tasks, it achieves an average 2.1× throughput improvement and 47% memory reduction during training and inference, while maintaining or exceeding baseline model accuracy. This enables effective low-resource deployment without compromising performance.
📝 Abstract
Recent progress in deep learning has been driven by increasingly larger models. However, their computational and energy demands have grown proportionally, creating significant barriers to their deployment and to a wider adoption of deep learning technologies. This thesis investigates principled approximation methods for improving the efficiency of deep learning systems, with a particular focus on settings that involve discrete constraints and non-differentiability.
We study three main approaches toward improved efficiency: architecture design, model compression, and optimization. For model compression, we propose novel approximations for pruning and quantization that frame the underlying discrete problem as continuous and differentiable, enabling gradient-based training of compression schemes alongside the model's parameters. These approximations allow for fine-grained sparsity and precision configurations, leading to highly compact models without significant fine-tuning. In the context of architecture design, we design an algorithm for neural architecture search that leverages parameter sharing across layers to efficiently explore implicitly recurrent architectures. Finally, we study adaptive optimization, revisiting theoretical properties of widely used methods and proposing an adaptive optimizer that allows for quick hyperparameter tuning.
Our contributions center on tackling computationally hard problems via scalable and principled approximations. Experimental results on image classification, language modeling, and generative modeling tasks show that the proposed methods provide significant improvements in terms of training and inference efficiency while maintaining, or even improving, the model's performance.