MaDeL: Manifold-Decomposed Feature Losses for Generative Modeling

2026-10-03

AI Summary

This study addresses the limitation of existing isotropic loss functions, which fail to distinguish valid intra-manifold variations from invalid out-of-manifold deviations. To overcome this, we propose an anisotropic learning framework that operates without explicit manifold estimation. By integrating feature bottlenecks with complementary representation learning, the method extracts complementary features from corrupted observations. Furthermore, it leverages Jacobian alignment to decouple tangent and normal spaces, directly separating parallel and orthogonal displacements from data to construct an anisotropic objective function capable of differentiating intrinsic variations from extrinsic deviations. Experimental results demonstrate that our approach significantly improves support set recovery accuracy and angular distance metrics while substantially reducing steric clashes in protein backbone structures.

Abstract

Generative models are often trained with isotropic objectives such as mean-squared error. For data concentrated near a low-dimensional manifold, however, such losses conflate displacement along the manifold, which may represent valid variation, with displacement away from it, which produces invalid samples. This mismatch is especially problematic in sparse, highly constrained domains, where ambient-space regression can encourage off-manifold interpolation. We ask whether a generative objective can distinguish manifold-parallel variation from manifold-orthogonal deviation directly from data, without explicitly estimating the manifold. We introduce a manifold-decomposed feature loss (MaDeL) that learns complementary representations from corrupted observations: one is trained to recover the clean sample, while the other is trained to recover the corruption. We show that, under a feature bottleneck, their Jacobians align with the tangent and normal spaces, exactly for linear manifolds and locally for smooth manifolds. Together, these representations define an anisotropic objective that separately measures intrinsic variation and off-manifold deviation. Across synthetic, Earth and climate science, and torsion-angle benchmarks, MaDeL improves support recovery and average angular $W_1$ under single-step sampling; on protein backbones, it reduces steric clashes across one- and few-step sampling budgets.

Problem

• Generative Modeling
• Manifold-Decomposed Loss
• Isotropic Objectives
• Low-Dimensional Manifold
• Anisotropic Objective

Innovation

• Manifold-Decomposed Loss
• Generative Modeling
• Anisotropic Objective
• Jacobian Alignment
• Feature Bottleneck