🤖 AI Summary
This work proposes a sparse and unified theoretical framework to systematically uncover the core mechanisms underlying learning, optimization, and modeling. It conceptualizes learning as a multi-level process arising from the coupling of problem formulation, method selection, and optimization dynamics. By precisely defining “solvable problems” and “parameterized methods,” the framework reduces complex learning theory to a few fundamental concepts rooted in dynamical systems, differential geometry, and foundational physics. The approach yields a general convergence theorem and establishes a universal theoretical foundation for cross-domain modeling and algorithm design, substantially enhancing both the parsimony and explanatory power of learning theory.
📝 Abstract
Nonlinear models and optimization methods have successfully tackled a rapidly growing set of problems in recent years. Indeed, a relatively small toolbox of such models and methods can provide sufficient performance across a large landscape of tasks: deep learning alone has made significant recent contributions in scientific modelling, natural language processing, visual analysis, etc. A similar relationship exists between physical theories and phenomena, where many applications and observations emerge neatly from remarkably minimal foundations. It is natural to wonder if sparse unified frameworks could be built to steer discussion and discovery in the fields concerned with learning, optimization, and modelling. In this work, we posit and examine a possible outline for such a unified theory, interpreting the notion of ''learning'' in a broad sense. In particular, we pursue our goals by viewing learning as an inter-connected process on multiple levels: problem setup, choosing methods, and the analysis of their interplay via imposed optimisation dynamics. We begin by proposing a precise yet versatile definition for ''solvable'' problems. We then define the ''parametrised methods'' by which their solution(s) may be ''learned''. Our goal is to sketch a ''universal convergence theorem'', specifying how and when solvable problems become amenable to the methods chosen for them. We find these constructions reduce the study of learning down to remarkably few ideas and tools - many of which are simply adapted from existing ones in dynamical systems theory, geometry, and fundamental physics.