🤖 AI Summary
This work addresses the longstanding divide between program execution and gradient-based optimization, which has hindered the use of general-purpose code as learnable scientific models. The authors propose the Differentiable Meta-Circular Interpreter (DMCI), which compiles a self-hosted subset of Scheme into a differentiable computation graph, enabling exact reverse-mode automatic differentiation of arbitrary recursive, higher-order programs with closures and complex data structures—without requiring recompilation or custom gradient definitions. This approach facilitates joint optimization over both program structure and parameters. Validated on 171 programs for gradient correctness, DMCI significantly outperforms handcrafted models and gradient-free methods in inverse problems such as battery degradation and El Niño prediction, successfully achieving symbolic regression and extrapolation with stateful executable programs.
📝 Abstract
The boundary between program execution and gradient-based optimization has long limited the use of code itself as a learnable scientific model. We present a compiler that translates a self-hosting subset of Scheme into differentiable computation graphs for autograd backends. Because the subset can compile its own evaluator, this yields differentiable meta-circular interpretation (DMCI): a compiled Scheme interpreter executes programs supplied as data, while reverse-mode autodiff propagates gradients to continuous constants embedded in those programs. The interpreter is compiled once, so new programs inherit differentiability without recompilation or custom gradient machinery, while retaining closures, recursion, and data structures. We prove that gradients through the compiled interpreter are correct almost everywhere and show that they match direct compilation to numerical precision across 171 recursive and higher-order program-seed pairs. We then use DMCI for program-and-parameter co-search, where a large language model proposes Scheme programs and exact gradients calibrate their continuous parameters through a single frozen interpreter. This enables OpenEvolve-style program search in which an outer loop proposes discrete program structures and DMCI supplies exact gradient-based calibration of each candidate's continuous parameters. On battery capacity-fade data, the search recovers a knee-like degradation structure and improves held-out extrapolation over hand-crafted baselines on the harder early-extrapolation split, matching them on the later split. On a high-dimensional El Nino inverse problem, DMCI optimizes an interpreted Kalman-filter likelihood where gradient-free search fails. These results extend symbolic regression and neurosymbolic search from closed-form expressions to executable, stateful programs, making model-generated code directly optimizable against data.