🤖 AI Summary
This study addresses the inherent non-identifiability of latent variables in factor models—manifested as non-uniqueness and distributional shifts—by systematically elucidating their nature in linear factor models and their implications for representation learning, drawing on an interdisciplinary perspective spanning psychometrics, statistics, and artificial intelligence. It establishes a theoretical connection between this identifiability issue and posterior collapse in variational autoencoders. By integrating factor analysis, linear autoencoders, and variational inference within a high-dimensional asymptotic framework, the work proves that latent factors become fully identifiable as the observation dimension tends to infinity. Building on this result, the authors propose a nearly distribution-free estimation method for high-dimensional settings, effectively bridging the theoretical gap between classical factor analysis and modern deep generative models, particularly well-suited for representation learning with ultra-high-dimensional data.
📝 Abstract
The common factor analytic model is related to Helmholtz and Boltzmann machines, can be conceived as a linear autoencoder, or can be thought of as a single-hidden-layer generative neural network. We thus consider it a basal generative representation learner that can be used as a minimal model for studying the foundational characteristics of (deep) generative model architectures. We focus on the fundamental problem of indeterminacy in latent factor projections. This indeterminacy implies that, even when the intrinsic dimension of the latent vector is known, regularity conditions are met, and rotational indeterminacy is resolved, an inherent indefiniteness in the retrieval of causative latent sources remains: they will be uncertain, distributionally deviant, and non-unique. This can have major implications for data representation but remains an elusive issue, even to practitioners and theorists well-versed in the factor model. Moreover, this classic psychometric problem is intricately related to the modern issue of latent variable collapse in the variational autoencoder framework for deep generative modeling. Here, we assess this indeterminacy from various perspectives and show how these are mathematically and conceptually related and we discuss subsequent implications for the Psychometrics, Statistics, and Artificial Intelligence communities. We show that one has latent factor determinacy across all its facets when the feature-dimension grows to infinity. This feeds into an essentially distribution-free estimation approach in the sample case when the number of features grows very large. We conclude, as these are emergent properties at scale, that the factor model is suited for representation learning of very-high-dimensional data.