What is the Relationship between Tensor Factorizations and Circuits (and How Can We Exploit it)?

2024-09-12

arXiv.org

AI Summary

This work bridges the theoretical gap between tensor decomposition and probabilistic circuit representations. Method: We propose a unified hierarchical factorization framework that establishes the first rigorous mathematical correspondence between classical tensor decompositions (e.g., CP, Tucker, Tensor Train) and structured probabilistic circuits, formally defining the class of “tensorizable circuits.” Our approach introduces a modular, composable circuit construction paradigm—akin to LEGO bricks—that enables differentiable architecture search and joint tensor-circuit parameter learning. Contribution/Results: Theoretically, we unify tensor-based and circuit-based probabilistic modeling under a common formalism. Methodologically, we present the first circuit learning framework supporting embedded multi-type tensor decompositions. Empirically, the framework demonstrates efficiency, interpretability, and scalability on probabilistic modeling tasks—including density estimation and latent-variable inference—thereby pioneering the new research direction of tensorized probabilistic circuits.

Technology Category

• Machine Learning: Probabilistic Circuits and Graphical Models
• Reasoning under Uncertainty: Relational Probabilistic Models
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization

Abstract

This paper establishes a rigorous connection between circuit representations and tensor factorizations, two seemingly distinct yet fundamentally related areas. By connecting these fields, we highlight a series of opportunities that can benefit both communities. Our work generalizes popular tensor factorizations within the circuit language, and unifies various circuit learning algorithms under a single, generalized hierarchical factorization framework. Specifically, we introduce a modular"Lego block"approach to build tensorized circuit architectures. This, in turn, allows us to systematically construct and explore various circuit and tensor factorization models while maintaining tractability. This connection not only clarifies similarities and differences in existing models, but also enables the development of a comprehensive pipeline for building and optimizing new circuit/tensor factorization architectures. We show the effectiveness of our framework through extensive empirical evaluations, and highlight new research opportunities for tensor factorizations in probabilistic modeling.

Problem

• Tensor Decomposition
• Circuit Representation
• Probabilistic Modeling

Innovation

• Tensor Decomposition
• Circuit Representation
• Systematic Framework