DIRESA, a distance-preserving nonlinear dimension reduction technique based on regularized autoencoders

2024-04-28

arXiv.org

AI Summary

To address the high computational and storage costs associated with online storage, transmission, and similarity-based retrieval of meteorological and climate big data, this paper proposes DIRESA: a Distance-regularized Siamese autoencoder framework. DIRESA simultaneously compresses nearline climate data while rigorously preserving the physical interpretability and ordinal relationships of distances in the latent space—balancing nonlinear representation capability, physical interpretability, latent variable disentanglement, and reconstruction fidelity. Unlike conventional linear (e.g., PCA) or unsupervised nonlinear dimensionality reduction methods (e.g., UMAP, VAE), DIRESA is the first to integrate pairwise distance regularization into a Siamese architecture, enabling end-to-end semantic-aware compression. Extensive evaluation across multiple complex climate models demonstrates that DIRESA’s latent variables exhibit clear physical meaning, achieves significantly higher storage compression ratios, and outperforms baseline methods in both distance preservation and reconstruction accuracy.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Data Mining & Knowledge Management: Data Compression
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics

Abstract

In meteorology, finding similar weather patterns or analogs in historical datasets can be useful for data assimilation, forecasting, and postprocessing. In climate science, analogs in historical and climate projection data are used for attribution and impact studies. However, most of the time, those large weather and climate datasets are nearline. This means that they must be downloaded, which takes a lot of bandwidth and disk space, before the computationally expensive search can be executed. We propose a dimension reduction technique based on autoencoder (AE) neural networks to compress the datasets and perform the search in an interpretable, compressed latent space. A distance-regularized Siamese twin autoencoder (DIRESA) architecture is designed to preserve distance in latent space while capturing the nonlinearities in the datasets. Using conceptual climate models of different complexities, we show that the latent components thus obtained provide physical insight into the dominant modes of variability in the system. Compressing datasets with DIRESA reduces the online storage and keeps the latent components uncorrelated, while the distance (ordering) preservation and reconstruction fidelity robustly outperform Principal Component Analysis (PCA) and other dimension reduction techniques such as UMAP or variational autoencoders.

Problem

• Compress large weather and climate datasets efficiently
• Preserve distance in latent space for accurate analogs search
• Improve interpretability and reduce storage compared to PCA

Innovation

• Autoencoder-based dimension reduction for datasets
• Distance-regularized Siamese twin autoencoder (DIRESA) architecture
• Preserves distance and nonlinearities in latent space