🤖 AI Summary
Standard backpropagation through time (BPTT) and its truncated or higher-order approximations suffer from significant gradient bias and unstable convergence in RNNs due to long-range dependencies. To address this, we propose an exact backward propagation method grounded in discrete forward sensitivity equations (DFSE). This is the first work to integrate DFSE into RNN training, enabling unbiased, full-sequence gradient computation while natively supporting time-varying parameters and multi-cycle coupled architectures. By performing precise Jacobian chain propagation, our method eliminates truncation errors and avoids cumulative bias from higher-order approximations. Experiments on long-sequence tasks demonstrate substantial improvements in gradient accuracy and training stability. Our approach establishes a new paradigm for modeling strong long-term dependencies in recurrent systems.
📝 Abstract
Backpropagation through time (BPTT) is a technique of updating tuned parameters within recurrent neural networks (RNNs). Several attempts at creating such an algorithm have been made including: Nth Ordered Approximations and Truncated-BPTT. These methods approximate the backpropagation gradients under the assumption that the RNN only utilises short-term dependencies. This is an acceptable assumption to make for the current state of artificial neural networks. As RNNs become more advanced, a shift towards influence by long-term dependencies is likely. Thus, a new method for backpropagation is required. We propose using the 'discrete forward sensitivity equation' and a variant of it for single and multiple interacting recurrent loops respectively. This solution is exact and also allows the network's parameters to vary between each subsequent step, however it does require the computation of a Jacobian.