Correlational Training of Morphological Neural Networks

2026-10-08

AI Summary

This study addresses the challenge that morphological neural networks suffer from sparse weight Jacobian matrices, which renders conventional gradient descent training ineffective. To overcome the limitations of backpropagation, this work proposes a non-gradient optimization strategy based on correlation rewards. Specifically, it introduces a Multiplicative Weight Update (MWU) scheme that formulates perceptron learning as a prediction-with-expert-advice problem, leveraging the alignment between inputs and desired output variations to drive weight updates in logarithmic space. Experimental evaluations across nine benchmarks demonstrate performance improvements in eight cases, with gains reaching up to 32.84 percentage points. Furthermore, the proposed method substantially reduces inter-run variance, effectively resolving the network training difficulties inherent in sparse gradient scenarios.

Abstract

Neural networks are typically trained using first-order methods and back-propagation. It is unclear whether this approach is optimal for morphological layers whose weight Jacobians are sparse and whose resulting parameter gradients can be poor. In this work, we propose a novel weight update method for morphological neural networks inspired from the Multiplicative Weights Update (MWU) scheme. We view each morphological perceptron as an instance of the learning from experts' advice problem in logarithmic space, and use a correlation-based reward that favors inputs aligned with the desired output change, regardless of whether a strong gradient signal has reached their weight. We empirically evaluate our approach by training fully connected layers both as stand-alone models and as parts of larger transformer networks. Across nine benchmarks, correlational training yields improvements on eight, by up to 32.84 percentage points, while substantially reducing run-to-run variability.

Problem

• Morphological Neural Networks
• Training Optimization
• Sparse Jacobians
• Poor Gradients

Innovation

• Morphological Neural Networks
• Multiplicative Weights Update
• Correlational Training
• Learning from Experts' Advice
• Back-propagation Alternative