Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

2023-10-31

arXiv.org

AI Summary

Existing deep learning pedagogy suffers from a disconnect between theory and practice, outdated mathematical tools, and insufficient theoretical grounding for modern architectures and optimization methods. Method: We propose a unified mathematical framework centered on *neural network calculus*—a rigorous differential calculus for neural networks—to systematically derive approximation capabilities of MLPs, CNNs, and RNNs; integrate the Kurdyka–Łojasiewicz framework to analyze convergence of SGD and its adaptive variants; formalize generalization error via precise mathematical characterization; and, for the first time, unify ANN calculus with physics-informed neural networks (PINNs) and deep Galerkin methods to enhance theoretical rigor and numerical stability in PDE solving. Contribution/Results: This work bridges foundational mathematics and practical deep learning, delivering a pedagogically viable, research-oriented paradigm that advances both teaching fidelity and theoretical depth—offering a mathematically principled, yet computationally grounded, foundation for next-generation deep learning.

Abstract

This book aims to provide an introduction to the topic of deep learning algorithms. We review essential components of deep learning algorithms in full mathematical detail including different artificial neural network (ANN) architectures (such as fully-connected feedforward ANNs, convolutional ANNs, recurrent ANNs, residual ANNs, and ANNs with batch normalization) and different optimization algorithms (such as the basic stochastic gradient descent (SGD) method, accelerated methods, and adaptive methods). We also cover several theoretical aspects of deep learning algorithms such as approximation capacities of ANNs (including a calculus for ANNs), optimization theory (including Kurdyka-{L}ojasiewicz inequalities), and generalization errors. In the last part of the book some deep learning approximation methods for PDEs are reviewed including physics-informed neural networks (PINNs) and deep Galerkin methods. We hope that this book will be useful for students and scientists who do not yet have any background in deep learning at all and would like to gain a solid foundation as well as for practitioners who would like to obtain a firmer mathematical understanding of the objects and methods considered in deep learning.

Problem

• Mathematical foundations of deep learning
• Different neural network architectures
• Deep learning for PDEs

Innovation

• Mathematical analysis of neural networks
• Optimization methods for deep learning
• Deep learning approaches for PDEs