Self-Rectifying Diffusion Sampling with Perturbed-Attention Guidance

Recent studies have demonstrated that diffusion models are capable of generating high quality samples, but their quality heavily depends on sampling guidance techniques, such as classifier guidance (CG) and classifier-free guidance (CFG). These techniques are often not applicable in unconditional generation or in various downstream tasks such as image restoration. In this paper, we propose a novel sampling guidance, called Perturbed-Attention Guidance (PAG), which improves diffusion sample quality across both unconditional and conditional settings, achieving this without requiring additional training or the integration of external modules. 

https://paperswithcode.com/paper/self-rectifying-diffusion-sampling-with

Papers with Code - Self-Rectifying Diffusion Sampling with Perturbed-Attention Guidance

 

PAG is designed to progressively enhance the structure of samples throughout the denoising process.

 

 PAG is designed to progressively enhance the structure of synthesized samples throughout the denoising process by considering the self-attention mechanisms' ability to capture structural information. It involves generating intermediate samples with degraded structure by substituting selected self-attention maps in diffusion U-Net with an identity matrix, and guiding the denoising process away from these degraded samples.

https://ku-cvlab.github.io/Perturbed-Attention-Guidance/

https://arxiv.org/pdf/2403.17377v1.pdf

In the section marked by the red rectangle in Fig. 3, it is apparent that the perturbed prediction (row 3 in (b)) is missing salient features like eyes, nose, and tongue. 

 

In this work, we proposed a novel guidance method, termed Perturbed-Attention Guidance (PAG), which leverages structural perturbation for improved image Self-Rectifying Diffusion Sampling with Perturbed-Attention Guidance 17 generation. 

Crucially, PAG achieves superior sample quality in both conditional and unconditional settings, requiring no additional training or external modules.

https://colab.research.google.com/drive/1GTRPAQZrnCDODL5Y8c8RUKbsY5SndEwe#scrollTo=dsU4sMPjiDUV

Google Colaboratory Notebook