TrioPose: Native Triple-Stream Diffusion Transformers for Pose-Guided Text-to-Image Generation

2026-06-05

AI Summary

This work addresses common issues of limb distortion and feature crosstalk in multi-person pose-guided text-to-image generation by introducing a native three-stream diffusion Transformer architecture built upon SD3.5M, which treats pose as an independent modality. The method incorporates hierarchical activation and zero-initialized dual residual injection to fuse geometric constraints, while a learnable relational bias mask disentangles occlusion interference. Additionally, heatmap-guided spatial loss weighting focuses optimization on deformation-prone regions. By preserving the stability of the pretrained latent space, the proposed approach significantly enhances generation accuracy, achieving state-of-the-art performance on Human-Art, CrowdPose, and OCHuman benchmarks. Notably, it attains an AP of 64.33 on Human-Art—a 30% improvement over prior methods—demonstrating markedly improved visual fidelity and semantic alignment.

Abstract

Pose-guided text-to-image generation often suffers from limb distortions and feature crosstalk in complex multi-person scenarios. While existing UNet-based adapters struggle with long-range spatial dependencies, emerging Multimodal Diffusion Transformers (MM-DiTs) offer superior global modeling. However, naive signal concatenation in MM-DiTs severely disrupts pre-trained latent distributions. To address this, we propose TrioPose, a native pose-driven framework built upon the SD3.5M architecture. Specifically, we introduce a Triple-Stream Pose-Aware DiT (TSPA-DiT) that treats pose as an independent modality. It employs layer-wise activation and zero-initialized dual-residual injection to smoothly enforce geometric constraints while preserving pre-trained latent stability. To resolve severe multi-instance occlusions, we design a Learnable Relational Bias Mask that categorizes topological connectivity into fine-grained physical states, mapping them into continuous attention soft constraints to effectively decouple inter-instance interference. Furthermore, a Pose-Guided Spatial Loss Weighting strategy modulates the native diffusion objective using heatmap-derived error maps, focusing anatomical supervision strictly on distortion-prone regions. Extensive experiments demonstrate that TrioPose achieves state-of-the-art performance across challenging benchmarks, including Human-Art, CrowdPose, and OCHuman. Notably, it attains an AP of $64.33$ on Human-Art, representing a $30\%$ improvement over prior arts, while setting new standards for visual fidelity and text-image semantic alignment in complex multi-human generation.

Problem

• pose-guided generation
• multi-person scenarios
• limb distortions
• feature crosstalk
• text-to-image generation

Innovation

• Triple-Stream Diffusion Transformer
• Pose-Guided Generation
• Learnable Relational Bias Mask
• Dual-Residual Injection
• Spatial Loss Weighting