Alice's Adventures in a Differentiable Wonderland - Volume I, A Tour of the Land

2024-04-26

arXiv.org

AI Summary

Novice learners often struggle to grasp the core concepts of neural networks and differentiable programming, exacerbated by a persistent disconnect between theoretical foundations and practical implementation. Method: This paper introduces a systematic pedagogical framework that unifies automatic differentiation, optimization fundamentals, and multimodal neural architectures—including CNNs, RNNs, and attention mechanisms—for sequences, graphs, text, and audio. Implemented natively in both PyTorch and JAX, the framework emphasizes self-contained, executable code to bridge the gap between mathematical derivation and engineering practice. Contribution/Results: It is the first to integrate large language models and multimodal modeling paradigms into differentiable programming education. The framework enables learners to deeply understand underlying principles while developing competencies in custom model design, debugging, and cross-modal transfer—thereby fostering both conceptual mastery and practical proficiency.

Technology Category

Machine Learning: Multimodal LearningNatural Language Processing: Learning & Optimization for NLPComputer Vision: Learning & Optimization for CV

Application Category

Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphsSearch and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingWeb Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Neural networks surround us, in the form of large language models, speech transcription systems, molecular discovery algorithms, robotics, and much more. Stripped of anything else, neural networks are compositions of differentiable primitives, and studying them means learning how to program and how to interact with these models, a particular example of what is called differentiable programming. This primer is an introduction to this fascinating field imagined for someone, like Alice, who has just ventured into this strange differentiable wonderland. I overview the basics of optimizing a function via automatic differentiation, and a selection of the most common designs for handling sequences, graphs, texts, and audios. The focus is on a intuitive, self-contained introduction to the most important design techniques, including convolutional, attentional, and recurrent blocks, hoping to bridge the gap between theory and code (PyTorch and JAX) and leaving the reader capable of understanding some of the most advanced models out there, such as large language models (LLMs) and multimodal architectures.

Problem

• Introducing differentiable programming basics for neural networks
• Optimizing functions via automatic differentiation techniques
• Designing models for sequences, graphs, texts, and audios

Innovation

• Automatic differentiation for function optimization
• Convolutional, attentional, recurrent neural blocks
• PyTorch and JAX implementation for advanced models