🤖 AI Summary
Recurrent Neural Networks (RNNs) and their variants—despite shared sequential modeling objectives—exhibit significant heterogeneity in architecture, objective functions, and learning algorithms, leading to conceptual ambiguity and fragmented understanding.
Method: This survey introduces the first unified taxonomy grounded in three orthogonal dimensions: network architecture, training objective, and optimization algorithm. It systematically categorizes mainstream models—including LSTMs, convolutional recurrent networks, graph/tree-structured RNNs, higher-order RNNs, and memory-augmented architectures—while analyzing their interdependencies and evolutionary trajectories.
Contribution/Results: The work establishes a generalizable modeling paradigm for complex sequence, speech, and image tasks; clarifies the fundamental distinctions and synergies between recursive and recurrent paradigms; synthesizes representative applications in NLP, automatic speech recognition, and image understanding; and identifies emerging research directions—including differentiable neural architecture search, dynamic computation graphs, and neuro-symbolic integration.
📝 Abstract
In this paper, the branches of recursive and recurrent neural networks are classified in detail according to the network structure, training objective function and learning algorithm implementation. They are roughly divided into three categories: The first category is General Recursive and Recurrent Neural Networks, including Basic Recursive and Recurrent Neural Networks, Long Short Term Memory Recursive and Recurrent Neural Networks, Convolutional Recursive and Recurrent Neural Networks, Differential Recursive and Recurrent Neural Networks, One-Layer Recursive and Recurrent Neural Networks, High-Order Recursive and Recurrent Neural Networks, Highway Networks, Multidimensional Recursive and Recurrent Neural Networks, Bidirectional Recursive and Recurrent Neural Networks; the second category is Structured Recursive and Recurrent Neural Networks, including Grid Recursive and Recurrent Neural Networks, Graph Recursive and Recurrent Neural Networks, Temporal Recursive and Recurrent Neural Networks, Lattice Recursive and Recurrent Neural Networks, Hierarchical Recursive and Recurrent Neural Networks, Tree Recursive and Recurrent Neural Networks; the third category is Other Recursive and Recurrent Neural Networks, including Array Long Short Term Memory, Nested and Stacked Recursive and Recurrent Neural Networks, Memory Recursive and Recurrent Neural Networks. Various networks cross each other and even rely on each other to form a complex network of relationships. In the context of the development and convergence of various networks, many complex sequence, speech and image problems are solved. After a detailed description of the principle and structure of the above model and model deformation, the research progress and application of each model are described, and finally the recursive and recurrent neural network models are prospected and summarized.