Equilibrium Training of Energy-Based Models with Parallel Trajectory Tempering

2026-07-29

AI Summary

This work addresses the instability and limited reliability of energy-based models (EBMs) in scientific data generation, which often stem from slow mixing in Markov chain Monte Carlo sampling. The authors propose a novel training algorithm based on Parallel Tempering with Trajectories (PTT), introducing PTT into the EBM framework for the first time. By leveraging the continuity of optimization paths, the method enables equilibrium sampling throughout training without additional computational overhead, facilitates accurate estimation of burn-in times, yields high-quality equilibrium samples, and supports exact log-likelihood computation. Integrated with reservoir sampling, adaptive optimization, and persistent contrastive divergence, the approach significantly outperforms existing deep generative models on discrete tabular data, demonstrating superior robustness, stability, and sample quality—particularly in small-sample and multimodal settings—while effectively mitigating overfitting.

Abstract

Energy-Based Models (EBMs) provide an interpretable framework for generative modeling of scientific data, but poor Markov Chain Monte Carlo mixing often limits their reliability. We introduce a training algorithm based on Parallel Trajectory Tempering (PTT), which exploits the continuity of the optimization path to maintain equilibrium sampling throughout learning. This enables stable and fast training on highly multimodal and data-scarce scientific datasets. Combined with reservoir sampling and adaptive optimization, PTT has a computational cost comparable to Persistent Contrastive Divergence, making it a practical replacement for standard training methods. It also provides direct estimates of thermalization times, equilibrium samples from trained models, and accurate log-likelihoods at essentially no additional cost. Experiments on Restricted Boltzmann Machines show that PTT consistently outperforms existing EBM training approaches. On discrete tabular data, it also surpasses state-of-the-art deep generative models, yielding higher-quality samples and greater robustness to overfitting and limited data. Our results make equilibrium maximum-likelihood training of EBMs practical and computationally efficient.

Problem

• Energy-Based Models
• Markov Chain Monte Carlo
• poor mixing
• equilibrium sampling
• generative modeling

Innovation

• Energy-Based Models
• Parallel Trajectory Tempering
• Equilibrium Training
• MCMC Mixing
• Reservoir Sampling