Semantic Space Search Trajectory Networks

2026-07-27

AI Summary

This work addresses the limitation of traditional Search Trajectory Networks (STNs) in high-dimensional or continuous settings, where reliance on raw spatial discretization hinders applicability. The authors propose the first extension of STNs into semantic space by constructing semantic representations from model prediction vectors over a fixed set of samples. Semantic vectors are then discretized and aggregated into nodes via agglomerative clustering based on normalized Hamming distance, enabling unified visualization and comparison of training dynamics across diverse learning algorithms. Applying this framework reveals systematic structural differences between networks trained on real versus random labels: STNs derived from real labels exhibit denser, more efficient, and centralized topologies, effectively capturing the relationship between neural network generalization and the underlying semantic structure of data.

Abstract

Search Trajectory Networks (STNs) are a graph-based tool for visualizing and characterizing the behavior of optimization algorithms. STNs' reliance on discretization of the search space has largely confined them to low-dimensional or combinatorial settings. We introduce a methodology for constructing STNs in semantic spaces, defined as the space of a model's predictions on a fixed sample set. Our approach discretizes semantic vectors and aggregates them into network nodes via agglomerative clustering with complete linkage under a normalized Hamming distance. Since any predictor can be summarized by its semantic vector, this method enables comparison of learning dynamics across otherwise incomparable algorithm families. We apply semantic space STNs to classification and regression tasks solved using different machine learning algorithms, recovering known qualitative differences between them. Additionally, we use semantic space STNs to study neural network generalization by contrasting standard training with the label randomization regime of Zhang et al. (2017). The resulting STNs exhibit consistent structural differences, training on real labels produces denser, more efficient and more centralized graphs than training on shuffled labels. Together, our results show that semantic space STNs capture functional training dynamics arising from the interaction between learning algorithms and data, providing a tool for analyzing and comparing learning dynamics across machine learning models and training regimes.

Problem

• Search Trajectory Networks
• semantic space
• learning dynamics
• optimization algorithms
• model comparison

Innovation

• Semantic Space
• Search Trajectory Networks
• Agglomerative Clustering
• Normalized Hamming Distance
• Learning Dynamics