Causal Intervention Framework for Variational Auto Encoder Mechanistic Interpretability

2025-05-06

AI Summary

This work addresses the limited mechanistic interpretability of Variational Autoencoders (VAEs), specifically focusing on how semantic factors are encoded, processed, and disentangled in the latent space. To this end, we introduce causal intervention analysis into VAEs for the first time, proposing a multi-granularity intervention framework encompassing input manipulation, latent-space perturbation, activation patching, and causal mediation analysis. We further develop a “circuit motif” identification method and novel metrics—including causal effect strength—to distinguish univariate versus multivariate neurons and quantify modular organization. Evaluated on standard disentanglement benchmarks, our approach successfully identifies functional circuits and maps computational graphs to semantic causal graphs: FactorVAE achieves a disentanglement score of 0.084 and a mean causal effect of 4.59—significantly outperforming standard VAE and β-VAE baselines.

Technology Category

Machine Learning: Causal LearningReasoning under Uncertainty: CausalityComputer Vision: Interpretability, Explainability, and Transparency

Application Category

Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphsSemantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMsUser Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization

Abstract

Mechanistic interpretability of deep learning models has emerged as a crucial research direction for understanding the functioning of neural networks. While significant progress has been made in interpreting discriminative models like transformers, understanding generative models such as Variational Autoencoders (VAEs) remains challenging. This paper introduces a comprehensive causal intervention framework for mechanistic interpretability of VAEs. We develop techniques to identify and analyze"circuit motifs"in VAEs, examining how semantic factors are encoded, processed, and disentangled through the network layers. Our approach uses targeted interventions at different levels: input manipulations, latent space perturbations, activation patching, and causal mediation analysis. We apply our framework to both synthetic datasets with known causal relationships and standard disentanglement benchmarks. Results show that our interventions can successfully isolate functional circuits, map computational graphs to causal graphs of semantic factors, and distinguish between polysemantic and monosemantic units. Furthermore, we introduce metrics for causal effect strength, intervention specificity, and circuit modularity that quantify the interpretability of VAE components. Experimental results demonstrate clear differences between VAE variants, with FactorVAE achieving higher disentanglement scores (0.084) and effect strengths (mean 4.59) compared to standard VAE (0.064, 3.99) and Beta-VAE (0.051, 3.43). Our framework advances the mechanistic understanding of generative models and provides tools for more transparent and controllable VAE architectures.

Problem

• Develop causal intervention framework for VAE interpretability
• Identify and analyze circuit motifs in VAEs
• Quantify interpretability of VAE components via metrics

Innovation

• Causal intervention framework for VAE interpretability
• Targeted interventions at multiple network levels
• Metrics for causal effect and circuit modularity