🤖 AI Summary
Existing inference-time alignment methods operate along a single control axis, struggling to model the joint dependency between conditioning variables and latent states and exhibiting limited generalization. This work proposes PG-MAP, a framework that, without requiring any training, formulates alignment as a joint maximum a posteriori (MAP) or proximal energy optimization over both conditional variables and latent states. It introduces a forward-consistency coupling mechanism to enable cross-modal collaborative updates. PG-MAP unifies support for both diffusion and flow-matching models by integrating trajectory-level Gibbs-MAP sampling, proximal optimization, and frozen preference-reward guidance, adapting flexibly to diverse generative transport dynamics. Experiments demonstrate that PG-MAP significantly improves PickScore and Aesthetic scores on SD1.5 and SDXL, achieving 91.9% PickScore and 75.7% HPS win rate on SD3.5-medium, with human evaluations consistently outperforming strong baselines.
📝 Abstract
Inference-time alignment of pretrained text-to-image models is typically performed along a single control axis, such as classifier-free guidance, attention editing, or reward-based latent perturbations. This limitation prevents modeling joint dependencies between conditioning and latent variables and hinders transfer across generative transports. We propose PG-MAP, a training-free framework that formulates inference-time alignment as a trajectory-level Gibbs-MAP / proximal energy optimization over the conditioning $c$ and latent state $z_t$ via a forward-consistency coupling, optionally guided by a frozen preference reward. This joint formulation enables coordinated updates across modalities while remaining compatible with both diffusion and flow-matching models through transport-specific adaptations. Across diffusion backbones (SD~1.5, SDXL), PG-MAP consistently improves alignment metrics such as PickScore and Aesthetic, and can be effectively combined with tuned classifier-free guidance to achieve the strongest overall performance. On flow-matching models (SD3.5-medium), the framework reduces to a latent-only variant, achieving $\mathbf{91.9\%}$ PickScore and $75.7\%$ HPS win rates against a static baseline, with controlled experiments ruling out noise-related artifacts. Human evaluations further confirm consistent preference over strong baselines, including tuned CFG and compute-matched universal guidance. Finally, an oracle-routing analysis shows that the relative importance of conditioning and latent optimization depends on prompt types, surfacing further headroom that a per-prompt selector could exploit.