Reinforcement learning in densely recurrent biological networks

2025-08-13

AI Summary

Training highly recurrent biological neural networks—such as the *C. elegans* full connectome—in continuous action spaces poses significant challenges: gradient-based methods suffer from vanishing/exploding gradients, while pure evolutionary algorithms converge slowly in high-dimensional weight spaces. Method: We propose ENOMAD, a hybrid framework integrating global evolutionary search with local derivative-free optimization (Mesh Adaptive Direct Search, MADS), augmented by biologically inspired weight priors. ENOMAD preserves the original circuit topology while enabling targeted functional specialization. Contribution/Results: Experiments on diverse foraging tasks demonstrate that ENOMAD significantly outperforms both untrained connectomes and state-of-the-art training approaches. It avoids gradient-related instabilities, improves training efficiency and behavioral performance in high-dimensional recurrent networks, and establishes a novel paradigm for functionally adapting biologically interpretable neural architectures.

Technology Category

Machine Learning: Evolutionary LearningSearch and Optimization: Evolutionary ComputationCognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystemsGraph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphsSearch and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Training highly recurrent networks in continuous action spaces is a technical challenge: gradient-based methods suffer from exploding or vanishing gradients, while purely evolutionary searches converge slowly in high-dimensional weight spaces. We introduce a hybrid, derivative-free optimization framework that implements reinforcement learning by coupling global evolutionary exploration with local direct search exploitation. The method, termed ENOMAD (Evolutionary Nonlinear Optimization with Mesh Adaptive Direct search), is benchmarked on a suite of food-foraging tasks instantiated in the fully mapped neural connectome of the nematode emph{Caenorhabditis elegans}. Crucially, ENOMAD leverages biologically derived weight priors, letting it refine--rather than rebuild--the organism's native circuitry. Two algorithmic variants of the method are introduced, which lead to either small distributed adjustments of many weights, or larger changes on a limited number of weights. Both variants significantly exceed the performance of the untrained connectome (in what can be interpreted as an example of transfer learning) and of existing training strategies. These findings demonstrate that integrating evolutionary search with nonlinear optimization provides an efficient, biologically grounded strategy for specializing natural recurrent networks towards a specified set of tasks.

Problem

• Training highly recurrent networks in continuous action spaces
• Overcoming gradient issues in dense biological networks
• Optimizing natural neural circuits for specific tasks

Innovation

• Hybrid evolutionary-direct search optimization
• Biologically derived weight priors utilization
• Two algorithmic variants for weight adjustment