manifold regularization

To address the challenges of modeling non-Euclidean data and the lack of dedicated computational tools, this paper introduces Manify, an open-source Python library that systematically unifies non-Euclidean representation learning, manifold-based classification/regression, and curvature estimation within a single framework. Methodologically, it leverages differential geometry and manifold optimization, supporting product manifold embeddings, Riemannian gradient descent, geodesic interpolation, and curvature tensor estimation. Its key contributions are: (1) the first open-source toolkit enabling compositional modeling over multiple manifold types; (2) provision of reproducible examples, benchmark datasets, and comprehensive documentation; and (3) substantial reduction of barriers to research and application in non-Euclidean machine learning, thereby facilitating the practical adoption of manifold learning in machine learning and data analysis.

This study investigates the geometric structure of image manifolds induced by 3D object poses and their inter-class variations, aiming to explain the success of visual representation learning from a differential-geometric perspective. Method: We propose a novel framework integrating geometry-preserving manifold learning with Kendall shape theory: (i) modeling pose-induced image manifolds as smooth, nonlinear manifolds in latent space; and (ii) introducing a rigidity-invariant Kendall metric to enable shape-invariant quantification and clustering of manifold geometry. Contribution/Results: We systematically demonstrate, for the first time, that image manifolds of the same object class exhibit significant clustering in shape space—and crucially, the degree of such clustering correlates with model generalization performance. Our approach enables comparable geometric modeling across object classes, providing both theoretically grounded, geometrically interpretable principles and practical guidance for designing vision algorithms.

To address metric distortion in pullback Riemannian geometry and diffeomorphic modeling error in multimodal data, this paper proposes the Iso-Riemannian manifold learning framework. It introduces isometric constraints into normalized-flow-driven pullback geometric learning for the first time, jointly optimizing diffeomorphic mappings and their induced pullback metrics. By incorporating Jacobian-based geometric regularization and differentiable manifold optimization, the method explicitly balances mapping regularity and representational capacity. Evaluated on synthetic and real-world multimodal datasets, it significantly improves interpolation continuity, clustering consistency, and distance preservation in low-dimensional representations—reducing geometric error by 37% and enhancing interpretability of downstream tasks. The core contribution is the establishment of the first scalable and interpretable isometric Riemannian manifold learning paradigm.

This work addresses the joint problem of intrinsic dimension estimation and geometry-invariant embedding learning for nonlinear manifold-structured data. We propose an autoencoder framework incorporating orthogonality constraints on hidden-layer gradients. Methodologically, we establish, for the first time, a theoretical connection between gradient orthogonality in neural network latent spaces and the local tangent space dimension of the underlying manifold; this enables simultaneous intrinsic dimension estimation, learning of invertible embedding mappings, and construction of coordinate-invariant representations under local Lie group actions on low-dimensional submanifolds. Our key contribution lies in unifying gradient orthogonality with differential-geometric structure, thereby extending invariant representation learning to continuous group actions. Experiments on standard benchmarks demonstrate accurate intrinsic dimension estimation, disentangled representations, and robust group-invariant embeddings, validating both theoretical soundness and algorithmic robustness.

This work addresses the lack of a universal statistical interpretation for the manifold hypothesis—that high-dimensional data approximately reside on low-dimensional manifolds. We propose the Latent Metric Model (LMM), a generative framework grounded in fundamental statistical concepts: latent variables, variable dependence, and stationarity—providing the first unified statistical justification for the manifold assumption. Methodologically, LMM integrates neighborhood graph construction, spectral analysis, and an interpretable inference framework to enable unsupervised manifold discovery and geometric structure recovery under weak priors. Experiments demonstrate that complex manifold geometries naturally emerge from minimal statistical mechanisms; LMM significantly reduces reliance on hand-crafted priors on both synthetic and real-world datasets, while enabling interpretable reconstruction of manifold dimensionality, curvature, and coordinate systems.

This study addresses the challenging problem of fitting an unknown number of hyperplanes to data, which involves non-convexity, non-differentiability, and uncertainty in model order. To tackle these difficulties, the authors propose a two-stage unsupervised learning approach grounded in a unit-sphere manifold framework. In the first stage, they integrate Riemannian expectation-maximization with heavy-tailed kernel density estimation to robustly infer posterior probabilities. The second stage employs hard assignment annealing to obtain geometrically consistent local optima. Key innovations include a manifold optimization framework for handling non-convex constraints, a projection-based density estimation scheme for initialization, and the two-stage optimization strategy itself. Experimental results demonstrate that the proposed method significantly outperforms state-of-the-art baselines in both geometric accuracy and robustness.

Existing anomaly detection methods typically assume that normal data occupy a non-zero volume in the ambient space, overlooking their intrinsic geometric structure as lying on a low-dimensional manifold, which limits performance. This work proposes a novel manifold projection paradigm: learning a projection operator that maps inputs onto the manifold of normal samples and using the projection residual as the anomaly criterion. By avoiding explicit modeling of the degenerate data distribution, the approach prevents misclassifying rare yet normal instances and provides a unified explanation for both the effectiveness and failure modes of reconstruction-based methods. Extensive experiments demonstrate that the proposed framework significantly outperforms conventional boundary-learning approaches and achieves state-of-the-art results across multiple benchmarks compared to existing reconstruction-based models.

This work addresses how to uncover interpretable concept manifolds embedded within the stacked representations of language models. The authors propose Manifold Probe, a method that generalizes traditional linear probing to manifold probing by integrating supervised manifold learning with linear predictability analysis. This approach identifies continuous geometric structures in representation space corresponding to high-level concepts—such as time or space—and determines their encoding directions. Beyond merely detecting the presence of such concepts, the method enables causal intervention: manipulating activations along discovered manifold directions directly alters model behavior. Experiments on Llama 2-7B demonstrate that perturbing representations along the extracted temporal manifold significantly shifts the model’s generated outputs regarding the release years of cultural works, thereby validating both the interpretability and causal efficacy of the recovered manifolds.

This work addresses the neglect of geometric relationships between convolutional kernels and features in existing deep representation learning. It introduces a dual-manifold perspective, modeling convolutional layers as a coupled system of a kernel manifold and a data manifold, and proposes a lightweight module, KGFT, which derives a geometry-guided matrix from the kernel Gram matrix to explicitly reshape feature covariance structures, thereby enabling geometric information transfer from the kernel to the data manifold. Innovatively incorporating an Exploit/Explore dual-mode mechanism with a depth-aware scheduling strategy, the method adaptively modulates guidance strength—promoting alignment in shallow layers and encouraging diversity in deeper ones. Extensive experiments demonstrate consistent performance gains across ResNet, ViT, and LLaMA-7B on image classification and arithmetic reasoning tasks, confirming the approach’s generality and effectiveness.

This work addresses the limitation of conventional optimization methods that impose uniform manifold constraints across all Transformer modules, disregarding their distinct geometric preferences in weight space. To remedy this, the authors introduce the Manifold Muon optimizer during GPT-2 pretraining, proposing a module-specific geometric optimization strategy by assigning Stiefel manifold constraints to attention layers and DGram manifold constraints to MLP layers. Experimental results demonstrate that this differentiated configuration substantially enhances training stability and model performance. In contrast, uniform DGram constraints—or inappropriate manifold assignments—tend to induce singular value growth in attention weights, leading to softmax saturation and unstable optimization. The study thus reveals an asymmetric geometric preference among Transformer components and advocates for function-aware customization of optimization geometry.