🤖 AI Summary
This work addresses the inefficiency of joint program-and-parameter search in neuro-symbolic learning, where each candidate program requires separate parameter optimization. To overcome this bottleneck, the authors propose the Neural Differentiable Virtual Machine (NDVM), which applies automatic differentiation to the interpreter rather than individual programs. By decoupling symbolic structure from differentiable numeric state, NDVM preserves program dynamics while enabling precise backpropagation through execution traces. The design incorporates dense batched numeric buffers and runtime symbolic environment management, substantially amortizing evaluation overhead. Experiments demonstrate that NDVM achieves approximately 60× amortized speedup per-channel batching over baseline methods, exhibits near-linear multi-core scaling, and accelerates the discovery of high-quality solutions by roughly 24× under a fixed computational budget.
📝 Abstract
AI systems increasingly propose executable scientific models whose value depends on both their symbolic structure and their fitted continuous parameters. This makes parameter calibration the bottleneck of program-and-parameter co-search: an outer loop can generate thousands of candidate programs, but each needs an inner gradient-based optimization before it can be assessed. Staging each candidate into its own differentiable graph makes individual models fast but sacrifices the program-as-data property that keeps search fluid; interpreter-based approaches preserve programs as runtime data but pay interpreter overhead that dominates the numerical work.
We present the Native Differentiable Virtual Machine (NDVM), a runtime representation that differentiates executable programs without compiling each candidate into a separate graph. NDVM separates symbolic structure from differentiable numeric state: tags, symbols, environments, and control remain native runtime data, while numeric payloads live in dense batched buffers with exact reverse-mode gradients recorded along the realized execution trace, so one evaluator walk is amortized across large populations of parameter vectors. A locked cost model of a real differentiable self-hosted Scheme interpreter motivates the design. We realize NDVM as a native runtime with forward and gradient equivalence to the reference backend, about 60x per-lane batch amortization, near-linear multicore scaling, and two independent front ends. In fixed-budget co-search over LLM-proposed programs, NDVM reaches high-quality solutions about 24x sooner in wall-clock time, suggesting runtime differentiation as a practical systems foundation for scientific discovery workflows.