Is the reconstruction loss culprit? An attempt to outperform JEPA

2026-03-14

AI Summary

This study investigates the root causes of performance degradation in autoencoders under noisy conditions, revealing that the primary issue lies not in the reconstruction loss itself but rather in the asymmetry of the objective function and the bottleneck/compositional selection effects. To address this, the authors propose the Gated Predictive Autoencoder (GPA), which explicitly selects predictable components to emulate the beneficial feature selection mechanism observed in overparameterized PCA. Evaluated on a controlled “TV series” linear dynamical system, GPA demonstrates robust performance across varying noise levels, matching or even surpassing the Joint Embedding Predictive Architecture (JEPA) while consistently outperforming conventional reconstruction-based autoencoders.

Technology Category

Machine Learning: Deep Generative Models & AutoencodersComputer Vision: Generative Adversarial Networks (GANs) for VisionSearch and Optimization: Learning to Search

Application Category

Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphsSearch and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingEconomics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

We evaluate JEPA-style predictive representation learning versus reconstruction-based autoencoders on a controlled "TV-series" linear dynamical system with known latent state and a single noise parameter. While an initial comparison suggests JEPA is markedly more robust to noise, further diagnostics show that autoencoder failures are strongly influenced by asymmetries in objectives and by bottleneck/component-selection effects (confirmed by PCA baselines). Motivated by these findings, we introduce gated predictive autoencoders that learn to select predictable components, mimicking the beneficial feature-selection behavior observed in over-parameterized PCA. On this toy testbed, the proposed gated model is stable across noise levels and matches or outperforms JEPA.

Problem

• reconstruction loss
• JEPA
• predictive representation learning
• autoencoders
• noise robustness

Innovation

• gated predictive autoencoders
• predictive representation learning
• JEPA
• feature selection
• reconstruction loss