Variational autoencoders with latent high-dimensional steady geometric flows for dynamics

2024-10-14

arXiv.org

AI Summary

Partial differential equation (PDE) modeling often suffers from degenerate latent manifolds and poor out-of-distribution (OOD) generalization. Method: This paper proposes Riemannian Variational Autoencoder VAE-DLM, which introduces a dynamical latent manifold governed by a high-dimensional geometric gradient flow derived from steady-state PDEs. It enforces manifold non-degeneracy and large measure via novel eigenvalue-based regularization to avoid metric singularity. The architecture employs a tanh-MLP-based Riemannian VAE with physics-informed ELBO loss and solves the linear geometric flow via first-order automatic differentiation. Results: On multiple PDE solution datasets, VAE-DLM reduces OOD prediction error by 15–35% over standard VAEs while maintaining or improving reconstruction accuracy, demonstrating significantly enhanced robustness to subtle late-time dynamics and external perturbations.

Technology Category

Machine Learning: Deep Generative Models & AutoencodersComputer Vision: Diffusion Models for VisionIntelligent Robots: Learning & Optimization for ROB

Application Category

User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendationGraph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphsResponsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

We develop Riemannian approaches to variational autoencoders (VAEs) for PDE-type ambient data with regularizing geometric latent dynamics, which we refer to as VAE-DLM, or VAEs with dynamical latent manifolds. We redevelop the VAE framework such that manifold geometries, subject to our geometric flow, embedded in Euclidean space are learned in the intermediary latent space developed by encoders and decoders. By tailoring the geometric flow in which the latent space evolves, we induce latent geometric properties of our choosing, which are reflected in empirical performance. We reformulate the traditional evidence lower bound (ELBO) loss with a considerate choice of prior. We develop a linear geometric flow with a steady-state regularizing term. This flow requires only automatic differentiation of one time derivative, and can be solved in moderately high dimensions in a physics-informed approach, allowing more expressive latent representations. We discuss how this flow can be formulated as a gradient flow, and maintains entropy away from metric singularity. This, along with an eigenvalue penalization condition, helps ensure the manifold is sufficiently large in measure, nondegenerate, and a canonical geometry, which contribute to a robust representation. Our methods focus on the modified multi-layer perceptron architecture with tanh activations for the manifold encoder-decoder. We demonstrate, on our datasets of interest, our methods perform at least as well as the traditional VAE, and oftentimes better. Our methods can outperform this and a VAE endowed with our proposed architecture, frequently reducing out-of-distribution (OOD) error between 15% to 35% on select datasets. We highlight our method on ambient PDEs whose solutions maintain minimal variation in late times. We provide empirical justification towards how we can improve robust learning for external dynamics with VAEs.

Problem

• Object Movement Analysis
• Complex Data Handling
• Error Rate Reduction

Innovation

• VAE-DLM
• Riemannian Geometry
• Gradient Mimicry