Same Loss, Different Gradients

2026-09-29

AI Summary

This study addresses the optimization trajectory deviation in probabilistic objective optimization caused by numerical inconsistencies between forward loss evaluation and backward gradient computation. To resolve this, it reveals that finite recursion induces non-conservative update fields and proposes fixed-depth analytical approximations via autoregressive (AR) and forward recursion (FR) schemes. Furthermore, this work introduces a novel differentiable primitive paradigm with joint value-derivative constraints, ensuring strict forward-backward alignment through the construction of potential functions and their derivatives. The resulting framework establishes uniform cubic error bounds across all axes, achieving improved precision with increasing dimensionality at constant computational depth. Applied to von Mises–Fisher distribution learning, this approach unifies certified fidelity with optimization consistency, transcending the limitations of traditional evaluations that focus solely on numerical accuracy.

Abstract

Differentiable learning typically assumes that the scalar objective evaluated in the forward pass and the gradient supplied to the optimizer in the backward pass describe the same mathematical object. We show that this correspondence can fail when probabilistic objectives rely on finite special-function recurrences, custom backward rules, and numerical clipping. In high-dimensional von Mises-Fisher learning, real numerical implementations can produce identical forward scores and losses at the same learning state while supplying different gradients and following different optimization trajectories. We characterize the structure of this mismatch in finite-start Bessel recurrence and show that classwise radial mismatch can compose through probabilities into a locally nonconservative update field. Evaluating the accuracy of special-function values and derivatives separately is therefore insufficient to characterize the realized learning objective. Motivated by this observation, we introduce AR/FR, a fixed-depth analytic realization that constructs a potential and its derivative jointly, ensuring forward-backward coherence by construction. We establish a uniform cubic-order error bound relative to the exact Bessel ratio over the entire nonnegative concentration axis and propagate this guarantee to learning scores and objectives. As representation dimension increases, the original finite recurrence becomes sequentially deeper, whereas the worst-case AR/FR error guarantee tightens cubically, jointly providing coherence, certified fidelity, and fixed-depth computation. These results suggest that a differentiable numerical primitive is defined by both the values it realizes and the derivatives it actually supplies to the optimizer; together, they constitute the numerical realization of the learning algorithm.

Problem

• differentiable learning
• gradient mismatch
• von Mises-Fisher
• Bessel recurrence
• forward-backward coherence

Innovation

• von Mises-Fisher learning
• Bessel recurrence
• forward-backward coherence
• AR/FR
• error bound