🤖 AI Summary
This work addresses parameter inference and learning in parametric models when the data-generating distribution is unknown or intractable. We propose a novel replica-theoretic framework based on variational Gaussian approximation. Within the grand canonical ensemble, we defer data averaging and replace the conventional quenched average with an empirical average; crucially, the trial Hamiltonian parameters are determined adaptively via a variational principle, eliminating reliance on idealized distributional assumptions. Our key contribution is the explicit incorporation of fluctuation effects into the analysis—establishing, for the first time within the replica method, a rigorous correspondence with information criteria such as AIC and BIC, thereby quantifying how statistical fluctuations govern generalization performance. Theoretical analysis yields exact learning curves for linear regression, and empirical validation confirms the framework’s efficacy in scenarios where standard replica methods break down.
📝 Abstract
We revisit the replica method for analyzing inference and learning in parametric models, considering situations where the data-generating distribution is unknown or analytically intractable. Instead of assuming idealized distributions to carry out quenched averages analytically, we use a variational Gaussian approximation for the replicated system in grand canonical formalism in which the data average can be deferred and replaced by empirical averages, leading to stationarity conditions that adaptively determine the parameters of the trial Hamiltonian for each dataset. This approach clarifies how fluctuations affect information extraction and connects directly with the results of mathematical statistics or learning theory such as information criteria. As a concrete application, we analyze linear regression and derive learning curves. This includes cases with real-world datasets, where exact replica calculations are not feasible.