loss landscape analysis

This work addresses the limited expressivity of conventional parameter optimization paradigms by proposing direct optimization of metric tensor fields on fixed-topology manifolds, enabling data-driven dynamic evolution of model geometry. Methodologically, leveraging discrete differential geometry, the manifold is represented as a triangular mesh, with the metric parameterized via edge lengths and optimized efficiently using automatic differentiation. Theoretically, the framework establishes a profound analogy between metric optimization and the Einstein–Hilbert action in general relativity. Crucially, geometric complexity adapts automatically while preserving topology, thereby enhancing model expressivity and generalization and effectively mitigating overfitting. The resulting framework provides a novel paradigm for scientific modeling, robust representation learning, and geometric deep learning—unifying geometric reasoning with differentiable optimization in a principled, topology-preserving manner.

This work systematically uncovers the decisive role of problem geometry—specifically, the curvature of the constraint set and the structure of gradients—in governing the statistical-computational trade-offs of stochastic and online optimization algorithms. We introduce the first geometric measure quantifying the deviation of a constraint set from quadratic convexity, rigorously identifying the geometric origins of suboptimality in subgradient methods. We prove that diagonal-preconditioned SGD achieves minimax-optimal convergence rates under quadratic convex constraints. For non-Euclidean, non-quadratically-convex domains—such as ℓₚ-balls with p < 2—we establish tight convergence bounds for mirror descent and adaptive gradient methods, and uncover, for the first time, a precise correspondence between their convergence rates and the accuracy-computation trade-off in Gaussian sequence estimation. Our results provide geometric criteria for algorithm selection and unify the understanding of when nonlinear updates—e.g., via mirror descent—are necessary to attain statistical optimality.

This work proposes a sparse and unified theoretical framework to systematically uncover the core mechanisms underlying learning, optimization, and modeling. It conceptualizes learning as a multi-level process arising from the coupling of problem formulation, method selection, and optimization dynamics. By precisely defining “solvable problems” and “parameterized methods,” the framework reduces complex learning theory to a few fundamental concepts rooted in dynamical systems, differential geometry, and foundational physics. The approach yields a general convergence theorem and establishes a universal theoretical foundation for cross-domain modeling and algorithm design, substantially enhancing both the parsimony and explanatory power of learning theory.

Neural network training often suffers from poor generalization and dynamic instability due to ill-conditioned loss landscapes—e.g., sharp minima and fractal structures. To address this, we propose the first optimizer grounded in rigid-body dynamics: parameter updates are modeled as frictionless rolling of a sphere over the loss manifold, with sphere radius serving as a learnable hyperparameter that enables intrinsic multi-scale geometric awareness and naturally suppresses sensitivity to local sharpness. Our method integrates continuous-time dynamical modeling, Riemannian differential geometry, and a gradient correction mechanism. Experiments across diverse architectures and datasets demonstrate substantial improvements in generalization and robustness, marked reduction in overfitting, enhanced training stability, and provision of an interpretable, multi-scale characterization of the loss terrain.

This work investigates the high-dimensional geometric structure of the solution space achieving zero training error in neural networks—modeled by binary-weight perceptrons—and its evolution with training set size. Using statistical physics methods—including the replica method, Gardner capacity analysis, and geometric characterization of solution spaces—we uncover a phase transition from dense, clustered solutions to sparse, isolated ones. We introduce “linear modal connectivity” as a quantitative measure of the average shape of solution manifolds. Crucially, we identify that algorithmic hardness arises from the disappearance of distant solution clusters precisely at the critical data threshold. Our analysis quantitatively characterizes the SAT/UNSAT phase transition, scaling laws of solution cluster sizes, and local landscape ruggedness. Collectively, these results establish a unified geometric–statistical physical framework for understanding generalization and optimization difficulty in deep learning.

This work addresses the limitations of conventional gradient-based optimization methods, which struggle to adapt to dynamic changes in length, curvature, and preconditioning implicitly induced by internal states under fixed geometric assumptions. The authors formulate optimization as a coupled system involving parameter trajectories, particle distributions, and a time-evolving Riemannian metric, explicitly distinguishing immutable obstacles from remediable geometric mismatches. They introduce the notion of “dynamic geometric complexity” and establish the first lower bound on geometric optimization difficulty based on affine-invariant distance. By leveraging gauge-invariant observables and Morse saddle-point flux analysis, they precisely characterize this complexity—in the setting of strongly convex quadratic objectives with a fully positive-definite metric oracle—as the affine-invariant distance from the relative logarithmic spectrum to the set of well-conditioned metrics.

This work addresses the limitations of traditional generalization analyses, which rely on the often unverifiable assumption of independent and identically distributed (i.i.d.) data and thus struggle to accurately characterize model performance on unseen data. The paper proposes a deterministic generalization analysis framework that dispenses with any prior probabilistic assumptions. By examining the sensitivity of optimization solutions to data perturbations, it decomposes the generalization error into geometric and probabilistic components, achieving their first-ever decoupling. The framework expresses generalization bounds via a variational principle, leveraging deterministic perturbation analysis and optimization sensitivity theory to capture the discrepancy between in-sample and out-of-sample performance. Error terms are evaluated through posterior statistical hypotheses, enabling the recovery of conventional high-probability or expected generalization guarantees—all without requiring distributional assumptions.

This work investigates the design of optimal curriculum strategies to enhance the efficiency and generalization of reinforcement learning. Inspired by nonequilibrium thermodynamics, it treats task reward parameters as coordinates on a task manifold and establishes, for the first time, a theoretical connection between curriculum learning and thermodynamic work: the optimal curriculum corresponds to a geodesic path in task space that minimizes excess work. Building upon this geometric principle, the paper proposes the Minimum Excess Work (MEW) algorithm, which integrates differential geometry, maximum-entropy reinforcement learning, and temperature annealing to automatically generate curricula with provable optimality in terms of temperature schedules. This approach provides a principled and computationally tractable geometric framework for curriculum generation.

This work addresses the challenge of achieving complete machine unlearning under second-order optimizers, where residual information from deleted data often persists. By modeling model memory through eigendecomposition and integrating state perturbation, geometric analysis, and counterfactual evaluation, the study systematically compares the unlearning behavior of first- and second-order learners. It reveals that although second-order optimizers can align gradients and performance with an ideal counterfactual model, their internal states retain latent, imperceptible memory traces. The authors demonstrate that only through carefully controlled perturbations that precisely erase geometric information can stable and effective forgetting be achieved. This work uncovers the geometric nature of memory in second-order optimization and establishes geometric information erasure as a critical mechanism for reliable machine unlearning.

This work investigates the role of projection heads in self-supervised learning, focusing on their impact on representation geometry, dimensional collapse, and the information–invariance trade-off. Modeling the projection head as a trainable Riemannian metric on the representation manifold induced by the backbone network, we offer the first differential-geometric characterization of its mechanism: nonlinear heads avoid representational collapse by inducing negative curvature, whereas linear or ReLU-based heads rely on BatchNorm and discrete optimization dynamics. Through Hessian spectral analysis, continuous-time gradient flows, and geometric tracking of optimization trajectories, we establish a link between metric degeneracy and information loss, and demonstrate that smooth activations such as Swish explicitly generate negative curvature. Our findings further reveal that projection heads act as universal geometric buffers and should be removed prior to downstream tasks to enhance performance.