Training Deep Visual Networks Beyond Loss and Accuracy Through a Dynamical Systems Approach

2026-04-08

AI Summary

This work addresses the limitations of conventional training metrics—such as loss and accuracy—in revealing the dynamic evolution of internal representations in deep visual networks. For the first time, it introduces signal dynamics methodologies from neuroscience into deep model analysis, adopting a dynamical systems perspective to construct three integrative metrics—integration, metastability, and composite stability—derived from layer-wise activation signals to characterize intrinsic training dynamics. Experiments across multiple mainstream architectures and CIFAR datasets demonstrate that integration effectively discriminates task difficulty, fluctuations in stability anticipate convergence behavior, and the relationship between integration and metastability reflects distinct training styles under different optimization strategies. These findings establish a novel evaluation framework that transcends traditional performance indicators, offering deeper insights into the mechanistic underpinnings of neural network training.

Technology Category

• Machine Learning: Evaluation and Analysis
• Search and Optimization: Metareasoning and Metaheuristics
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Deep visual recognition models are usually trained and evaluated using metrics such as loss and accuracy. While these measures show whether a model is improving, they reveal very little about how its internal representations change during training. This paper introduces a complementary way to study that process by examining training through the lens of dynamical systems. Drawing on ideas from signal analysis originally used to study biological neural activity, we define three measures from layer activations collected across training epochs: an integration score that reflects long-range coordination across layers, a metastability score that captures how flexibly the network shifts between more and less synchronised states, and a combined dynamical stability index. We apply this framework to nine combinations of model architecture and dataset, including several ResNet variants, DenseNet-121, MobileNetV2, VGG-16, and a pretrained Vision Transformer on CIFAR-10 and CIFAR-100. The results suggest three main patterns. First, the integration measure consistently distinguishes the easier CIFAR-10 setting from the more difficult CIFAR-100 setting. Second, changes in the volatility of the stability index may provide an early sign of convergence before accuracy fully plateaus. Third, the relationship between integration and metastability appears to reflect different styles of training behaviour. Overall, this study offers an exploratory but promising new way to understand deep visual training beyond loss and accuracy.

Problem

• deep visual networks
• internal representations
• training dynamics
• dynamical systems
• layer activations

Innovation

• dynamical systems
• integration score
• metastability
• layer activations
• training dynamics