Latent Diffusion U-Net Representations Contain Positional Embeddings and Anomalies

2025-04-09

AI Summary

This work identifies three structural deficiencies in the intermediate representations of the Stable Diffusion U-Net: (1) implicit positional embedding biases, (2) non-semantic, high-similarity artifacts at image corners, and (3) abnormally large L²-norm responses in specific channels. Using representational similarity analysis (RSA) and norm-based statistics—combined with multi-layer feature visualization and quantitative diagnostics across SD v1.4 and v2.1—we systematically uncover, for the first time, strong positional preferences and two reproducible structural anomalies in diffusion latent spaces. These findings challenge the implicit assumption that diffusion model intermediate features are inherently suitable for direct downstream use. We propose a novel three-dimensional credibility metric grounded in positional bias, corner similarity, and channel-wise norm statistics. This framework provides both theoretical grounding and empirical evidence for robust representation modeling and trustworthy deployment of diffusion-based features.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Representation Learning
• Reasoning under Uncertainty: Uncertainty Representations

Application Category

• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Diffusion models have demonstrated remarkable capabilities in synthesizing realistic images, spurring interest in using their representations for various downstream tasks. To better understand the robustness of these representations, we analyze popular Stable Diffusion models using representational similarity and norms. Our findings reveal three phenomena: (1) the presence of a learned positional embedding in intermediate representations, (2) high-similarity corner artifacts, and (3) anomalous high-norm artifacts. These findings underscore the need to further investigate the properties of diffusion model representations before considering them for downstream tasks that require robust features. Project page: https://jonasloos.github.io/sd-representation-anomalies

Problem

• Analyze positional embeddings in diffusion model representations
• Identify high-similarity corner artifacts in Stable Diffusion
• Investigate anomalous high-norm artifacts in model outputs

Innovation

• Analyze Stable Diffusion models using similarity and norms
• Identify learned positional embedding in representations
• Detect high-similarity and high-norm artifacts