Autoencoding Random Forests

2025-05-27

AI Summary

This paper introduces the first general-purpose, universally consistent autoencoder framework based on random forests, designed for learning low-dimensional embeddings and enabling invertible reconstruction—supporting downstream tasks including visualization, compression, clustering, and denoising. Methodologically, it integrates nonparametric statistics with spectral graph theory to learn embedding structure, and proposes three invertible decoding mechanisms: constrained optimization, tree-structure-based splitting and relabeling, and k-nearest-neighbor regression—thereby establishing a bidirectional mapping between input and embedding spaces. Unlike deep learning approaches, the framework requires no neural networks, accommodates both supervised and unsupervised settings, and permits analytical derivation of conditional or joint distributions. Experiments on tabular, image, and genomic datasets demonstrate that its decoder achieves theoretically guaranteed universal consistency, significantly enhancing model interpretability and improving performance across diverse downstream tasks.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Reasoning under Uncertainty: Graphical Models
• Computer Vision: Representation Learning for Vision

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

We propose a principled method for autoencoding with random forests. Our strategy builds on foundational results from nonparametric statistics and spectral graph theory to learn a low-dimensional embedding of the model that optimally represents relationships in the data. We provide exact and approximate solutions to the decoding problem via constrained optimization, split relabeling, and nearest neighbors regression. These methods effectively invert the compression pipeline, establishing a map from the embedding space back to the input space using splits learned by the ensemble's constituent trees. The resulting decoders are universally consistent under common regularity assumptions. The procedure works with supervised or unsupervised models, providing a window into conditional or joint distributions. We demonstrate various applications of this autoencoder, including powerful new tools for visualization, compression, clustering, and denoising. Experiments illustrate the ease and utility of our method in a wide range of settings, including tabular, image, and genomic data.

Problem

• Autoencoding with random forests for low-dimensional embeddings
• Inverting compression via optimization and nearest neighbors regression
• Applications in visualization, compression, clustering, and denoising

Innovation

• Autoencoding with random forests via spectral graph theory
• Decoding via constrained optimization and nearest neighbors
• Works with supervised or unsupervised models universally