Beyond Local Linearity: Scale-Resolved Geometry of Learned Image Encoders

2026-09-30

AI Summary

This study addresses the limitation of existing geometric analyses, which characterize only local perturbations and fail to capture encoder robustness to finite input variations. We propose scale-resolved statistics that contrast the evolution of feature displacements against local linear predictions across perturbation magnitudes, thereby revealing geometric deviations during learning. Our analysis uncovers a "hump" curve in feature space—a phenomenon absent in untrained models that emerges through standard training and depends on the data distribution. These findings establish non-local geometric deviation as an intrinsic signature of representation learning, offering a novel perspective for understanding the feature-shaping mechanisms of deep encoders.

Abstract

Understanding how learned representations respond to finite input changes is important for characterizing their sensitivity, invariances, and robustness. Yet existing geometric analyses are predominantly local and describe only infinitesimal perturbations. We introduce a scale-resolved statistic that compares an encoder's measured feature displacement with its local linear prediction as the perturbation magnitude increases. Across diverse image encoders, we discover a characteristic plateau-rise-peak-decay profile, which we call the bump. The bump is absent at initialization, emerges early during standard training, and does not form under randomized labels or random-noise inputs. Its shape also varies with the training distribution and robustness objective. These results establish departures from local geometry as a signature of how encoder representations are shaped by learning.

Problem

• learned representations
• image encoders
• local linearity
• finite perturbations
• geometric analysis

Innovation

• Scale-Resolved Geometry
• Local Linearity
• Image Encoders
• Feature Displacement
• Representation Learning