Dive deep into stable diffusion samplers.

We will learn what is a sampler in stable diffusion and basic maths involved in it. Sampler is a very essential topic in stable diffusion because we will use it in various text to image creation tools like AUTOMATIC1111 Stable Diffusion GUI and comfyUI etc.

You may see something like this ⬇️ while using automatic1111 or comyfyUI.

What Happens Behind the Sampler ?

The noise predictor model (model) will predict the noise from the text to image creation process, next the sampler contains a set of mathematic equations to perform sampling and denoising to create a final result. This is process occurs several times.

The mathematic equation looks like this

xt = αt/αs * xs − αt Z λt λs e −λ ϵˆθ(xˆλ, λ)dλ,

above equation is a ODE solver I didn't understand a single thing from the equation 😅

We're not going to learn that equation but You will understand how it works.

Sampling

We start with random noise and another type of noise that we want to remove, which we call predicted noise. This process is repeated several times to achieve a clear end result. In each iteration, the sampler modifies the image to improve it from the previous version, and this overall process is known as "sampling."

Sampling steps

Each individual modification or adjustment made by the sampler is referred to as a "sampling step."

Denoising

During each sampling step, the original noise is gradually reduced to create the final, cleaned-up image. This reduction of noise in each step is specifically termed "denoising."

Example of sampling

Sampler Methods

We will learn how each sampler method in the AUTOMATIC1111 Stable Diffusion GUI works and beyond it. I categorized the samplers for better understanding.

Fastest Methods

DPM++ 2M

It's the fastest and most advanced solver in the DPM family. It produces high-quality images in 14 to 20 steps, which is a bit slow compared to Euler but still faster than other solvers.

Break down the name

• D - Diffusion (Spreading out the noise)
• P - Probabilistic (Predicting)
• M - Model
• (++) - Improved version of DDIM sampler (Oldest sampler)
• 2 - Second order mathematic equation
• M - Multistep process involved

what makes it faster ?

It uses Multistep Variant to prevent instability(where the noise becomes unpredictable), Basically it takes smaller steps when solving the equation. Think of it like breaking down the problem into smaller, more manageable pieces to get a more stable and accurate result.

Example image

Prompt: red car driving in the highway, night background,4k

Model: SDXL base 1.0.safetensor

DPM++ SDE

My personal favorite because it is neither too slow nor too fast, but it serves the job well. It takes 10 to 20 steps to create a high-quality image. We will learn what is SDE later on this documentation.

Example image

Same prompt

Steps : 20 steps

Euler

The fastest and simplest diffusion solver in the list. It is a first-order solver, which means it does one computation with its takes its step.

To make more sense, think of it like a first order solvers being a person who can change strategies in each step to work at different level and speed depending on the workload but the second order solvers are the person who always works efficient and fast in all the time.

For better understand

When we use Euler and DPM++ 2M and set the step size to 15 for both, the first sampler takes random noise. With Euler, the process involves taking big jumps to obtain a clear image. On the other hand, DPM++2M jumps equal distance to achieve the result. However, if we increase the step size to 200, Euler takes small jumps in each steps, and by the 200th step, we get a clear result. In contrast, DPM++2M achieves a clear result in just middle steps. The second order solver is more stable and accurate than first order.

Still can't understand watch this video

Brock Webb. He talks about how the euler works.

Example image

Same prompt

Steps : 20 steps

LMS

A Linear Multi-Step method is like a smarter version of Euler's method. Instead of only looking at the immediate past step to predict the next one, it looks at several past steps. This helps make the predictions more accurate.

Example image

Stable Methods

Heun

Heun is like the update euler. It is half as fast because it is a second order solver and it uses the same calculation that euler does but it has second mathematic equation. Which provides more stable and better results than euler.

Example image

Restart

Yilun Xu and team came up with a new sampling algorithm. It works by doing something called "restarts," where it adds extra noise at intervals during the usual noise reduction process. According to the authors, this method combines the benefits of both deterministic and stochastic samplers. It's fast like deterministic samplers and doesn't get stuck in one place like stochastic samplers.

Example image

Ancestral sampler

Ancestral samplers is a addtional layer for each solvers, which is denoted as "A" at the end of the sampler names

For example

Euler A, DPM++ 2s A, DPM2 A

When to choose Ancestral samplers

It is more suitable for creating randomness images. It changes the image in each steps and the end results are also usually quite different from the non-ancestral counterpart, often regarded as more "creative".

Image converge

Mostly all the ancestral sampler are Image converge.

Here a example

In the above example of image converge, The bear's position is changing in each steps and the end result is way different.

In the above example of doesn't converge, The bear's position is changing but not like the Euler ancestral

karras Methods

You can identify the karras methods very easily because it has the word karras at the end.

For example

DPM++ 2M karras DPM++ 2S karras DPM++ SDE karras

Generally, it's effective to take big steps initially and smaller steps later on. The Karras schedule is a slight adjustment to the standard schedule, which has been found to work better based on practical experience.

Comparison

I added example images for all the samplers below. Maybe all look the same but they are slightly different from each others.

My Final Verdict

Verdict 1: I choose Euler for its simplicity and speed.

Verdict 2: If I want a stable and better quality image, then I prefer to use DPM++ 2M.

Always explore on your own, experiment with different parameters, and find what works best for you.