Diffusion model

Diffusion models were introduced in 2015 as a method to learn a model that can sample from a highly complex probability distribution. They used techniques from non-equilibrium thermodynamics, especially diffusion.[5]

Consider, for example, how one might model the distribution of all naturally-occurring photos. Each image is a point in the space of all images, and the distribution of naturally-occurring photos is a "cloud" in space, which, by repeatedly adding noise to the images, diffuses out to the rest of the image space, until the cloud becomes all but indistinguishable from a gaussian distribution ${\displaystyle N(0,I)}$. A model that can approximately undo the diffusion can then be used to sample from the original distribution. This is studied in "non-equilibrium" thermodynamics, as the starting distribution is not in equilibrium, unlike the final distribution.

The equilibrium distribution is the gaussian distribution ${\displaystyle N(0,I)}$, with pdf ${\displaystyle \rho (x)\propto e^{-{\frac {1}{2}}\|x\|^{2}}}$. This is just the Boltzmann distribution of particles in a potential well ${\displaystyle V(x)={\frac {1}{2}}\|x\|^{2}}$ at temperature 1. The initial distribution, being very much out of equilibrium, would diffuse towards the equilibrium distribution, making biased random steps that are a sum of pure randomness (like a Brownian walker) and gradient descent down the potential well. The randomness is necessary: if the particles were to undergo only gradient descent, then they will all fall to the origin, collapsing the distribution.

The 2020 paper proposed the Denoising Diffusion Probabilistic Model (DDPM), which improves upon the previous method by variational inference.[6]

To present the model, we need some notation.

• ${\displaystyle \beta _{1},...,\beta _{T}\in (0,1)}$ are fixed constants.
• ${\displaystyle \alpha _{t}:=1-\beta _{t}}$
• ${\displaystyle {\bar {\alpha }}_{t}:=\alpha _{1}\cdots \alpha _{t}}$
• ${\displaystyle {\tilde {\beta }}_{t}:={\frac {1-{\bar {\alpha }}_{t-1}}{1-{\bar {\alpha }}_{t}}}\beta _{t}}$
• ${\displaystyle {\tilde {\mu }}_{t}(x_{t},x_{0}):={\frac {{\sqrt {\alpha _{t}}}(1-{\bar {\alpha }}_{t-1})x_{t}+{\sqrt {{\bar {\alpha }}_{t-1}}}(1-\alpha _{t})x_{0}}{1-{\bar {\alpha }}_{t}}}}$
• ${\displaystyle N(\mu ,\Sigma )}$ is the normal distribution with mean ${\displaystyle \mu }$ and variance ${\displaystyle \Sigma }$, and ${\displaystyle N(x|\mu ,\Sigma )}$ is the probability density at ${\displaystyle x}$.
• A vertical bar denotes conditioning.

A forward diffusion process starts at some starting point ${\displaystyle x_{0}\sim q}$, where ${\displaystyle q}$ is the probability distribution to be learned, then repeatedly add noise to it by

$${\displaystyle x_{t}={\sqrt {1-\beta _{t}}}x_{t-1}+{\sqrt {\beta _{t}}}z_{t}}$$

where ${\displaystyle z_{1},...,z_{T}}$ are IID samples from ${\displaystyle N(0,I)}$. The entire diffusion process then satisfies

$${\displaystyle q(x_{0:T})=q(x_{0})q(x_{1}|x_{0})\cdots q(x_{T}|x_{T-1})=q(x_{0})N(x_{1}|{\sqrt {\alpha _{1}}}x_{0},\beta _{1}I)\cdots N(x_{T}|{\sqrt {\alpha _{T}}}x_{T-1},\beta _{T}I)}$$

or

$${\displaystyle \ln q(x_{0:T})=\ln q(x_{0})-\sum _{t=1}^{T}{\frac {1}{2\beta _{t}}}\|x_{t}-{\sqrt {1-\beta _{t}}}x_{t-1}\|^{2}+C}$$

where ${\displaystyle C}$ is a normalization constant and often omitted. In particular, we note that ${\displaystyle x_{1:T}|x_{0}}$ is a gaussian process, which affords us considerable freedom in reparametrization. For example, by standard manipulation with gaussian process,

$${\displaystyle x_{t}|x_{0}\sim N\left({\sqrt {{\bar {\alpha }}_{t}}}x_{0},(1-{\bar {\alpha }}_{t})I\right)}$$

$${\displaystyle x_{t-1}|x_{t},x_{0}\sim N({\tilde {\mu }}_{t}(x_{t},x_{0}),{\tilde {\beta }}_{t}I)}$$

In particular, notice that for large ${\displaystyle t}$, the variable ${\displaystyle x_{t}|x_{0}\sim N\left({\sqrt {{\bar {\alpha }}_{t}}}x_{0},(1-{\bar {\alpha }}_{t})I\right)}$ converges to ${\displaystyle N(0,I)}$. That is, after a long enough diffusion process, we end up with some ${\displaystyle x_{T}}$ that is very close to ${\displaystyle N(0,I)}$, with all traces of the original ${\displaystyle x_{0}\sim q}$ gone. For example, since

$${\displaystyle x_{t}|x_{0}\sim N\left({\sqrt {{\bar {\alpha }}_{t}}}x_{0},(1-{\bar {\alpha }}_{t})I\right)}$$

we can sample ${\displaystyle x_{t}|x_{0}}$ directly "in one step", instead of going through all the intermediate steps ${\displaystyle x_{1},x_{2},...,x_{t-1}}$.

The key idea of DDPM is to use a neural network parametrized by ${\displaystyle \theta }$. The network takes in two arguments ${\displaystyle x_{t},t}$, and outputs a vector ${\displaystyle \mu _{\theta }(x_{t},t)}$ and a matrix ${\displaystyle \Sigma _{\theta }(x_{t},t)}$, such that each step in the forward diffusion process can be approximately undone by ${\displaystyle x_{t-1}\sim N(\mu _{\theta }(x_{t},t),\Sigma _{\theta }(x_{t},t))}$. This then gives us a backward diffusion process ${\displaystyle p_{\theta }}$ defined by

$${\displaystyle p_{\theta }(x_{T})=N(x_{T}|0,I)}$$

$${\displaystyle p_{\theta }(x_{t-1}|x_{t})=N(x_{t-1}|\mu _{\theta }(x_{t},t),\Sigma _{\theta }(x_{t},t))}$$

The goal now is to learn the parameters such that ${\displaystyle p_{\theta }(x_{0})}$ is as close to ${\displaystyle q(x_{0})}$ as possible. To do that, we use maximum likelihood estimation with variational inference.

The ELBO inequality states that ${\displaystyle \ln p_{\theta }(x_{0})\geq E_{x_{1:T}\sim q(\cdot |x_{0})}[\ln p_{\theta }(x_{0:T})-\ln q(x_{1:T}|x_{0})]}$, and taking one more expectation, we get

$${\displaystyle E_{x_{0}\sim q}[\ln p_{\theta }(x_{0})]\geq E_{x_{0:T}\sim q}[\ln p_{\theta }(x_{0:T})-\ln q(x_{1:T}|x_{0})]}$$

We see that performing maximizing the quantity on the right would give us a lower bound on the likelihood of observed data. This allows us to perform variational inference. Define the loss function

$${\displaystyle L(\theta ):=-E_{x_{0:T}\sim q}[\ln p_{\theta }(x_{0:T})-\ln q(x_{1:T}|x_{0})]}$$

and now the goal is to minimize the loss by stochastic gradient descent. The expression may be simplified to[7]

$${\displaystyle L(\theta )=\sum _{t=1}^{T}E_{x_{t-1},x_{t}\sim q}[-\ln p_{\theta }(x_{t-1}|x_{t})]+E_{x_{0}\sim q}[D_{KL}(q(x_{T}|x_{0})\|p_{\theta }(x_{T}))]+C}$$

where ${\displaystyle C}$ does not depend on the parameter, and thus can be ignored. Since ${\displaystyle p_{\theta }(x_{T})=N(x_{T}|0,I)}$ also does not depend on the parameter, the term ${\displaystyle E_{x_{0}\sim q}[D_{KL}(q(x_{T}|x_{0})\|p_{\theta }(x_{T}))]}$ can also be ignored. This leaves just ${\displaystyle L(\theta )=\sum _{t=1}^{T}L_{t}}$ with ${\displaystyle L_{t}=E_{x_{t-1},x_{t}\sim q}[-\ln p_{\theta }(x_{t-1}|x_{t})]}$ to be minimized.

Since ${\displaystyle x_{t-1}|x_{t},x_{0}\sim N({\tilde {\mu }}_{t}(x_{t},x_{0}),{\tilde {\beta }}_{t}I)}$, this suggests that we should use ${\displaystyle \mu _{\theta }(x_{t},t)={\tilde {\mu }}_{t}(x_{t},x_{0})}$; however, the network does not have access to ${\displaystyle x_{0}}$, and so it has to estimate it instead. Now, since ${\displaystyle x_{t}|x_{0}\sim N\left({\sqrt {{\bar {\alpha }}_{t}}}x_{0},(1-{\bar {\alpha }}_{t})I\right)}$, we may write ${\displaystyle x_{t}={\sqrt {{\bar {\alpha }}_{t}}}x_{0}+{\sqrt {1-{\bar {\alpha }}_{t}}}z}$, where ${\displaystyle z}$ is some unknown gaussian noise. Now we see that estimating ${\displaystyle x_{0}}$ is equivalent to estimating ${\displaystyle z}$.

Therefore, let the network output a noise vector ${\displaystyle \epsilon _{\theta }(x_{t},t)}$, and let it predict

$${\displaystyle \mu _{\theta }(x_{t},t)={\tilde {\mu }}_{t}\left(x_{t},{\frac {x_{t}-{\sqrt {1-{\bar {\alpha }}_{t}}}\epsilon _{\theta }(x_{t},t)}{\sqrt {{\bar {\alpha }}_{t}}}}\right)={\frac {x_{t}-\epsilon _{\theta }(x_{t},t)\beta _{t}/{\sqrt {1-{\bar {\alpha }}_{t}}}}{\sqrt {\alpha _{t}}}}}$$

It remains to design ${\displaystyle \Sigma _{\theta }(x_{t},t)}$. The DDPM paper suggested not learning it (since it resulted it "unstable training and poorer sample quality"), but fixing it at some value ${\displaystyle \Sigma _{\theta }(x_{t},t)=\sigma _{t}^{2}I}$, where either ${\displaystyle \sigma _{t}^{2}=\beta _{t}{\text{ or }}{\tilde {\beta }}_{t}}$ yielded similar performance. With this, the loss simplifies to

$${\displaystyle L_{t}={\frac {\beta _{t}^{2}}{2\alpha _{t}(1-{\bar {\alpha }}_{t})\sigma _{t}^{2}}}E_{x_{0}\sim q;z\sim N(0,I)}\left[\left\|\epsilon _{\theta }(x_{t},t)-z\right\|^{2}\right]+C}$$

which may be minimized by stochastic gradient descent. The paper noted empirically that an even simpler loss function

$${\displaystyle L_{simple,t}=E_{x_{0}\sim q;z\sim N(0,I)}\left[\left\|\epsilon _{\theta }(x_{t},t)-z\right\|^{2}\right]}$$

resulted in better models.

Consider the problem of image generation. Let ${\displaystyle x}$ represent an image, and let ${\displaystyle q(x)}$ be the probability distribution over all possible images. If we have ${\displaystyle q(x)}$ itself, then we can say for certain how likely a certain image is. However, this is intractable in general.

Most often, we are uninterested in knowing the absolute probability of a certain image. Instead, we are usually only interested in knowing how likely a certain image is compared to its immediate neighbors — e.g. how much more likely is an image of cat compared to some small variants of it? Is it more likely if the image contains two whiskers, or three, or with some Gaussian noise added?

Consequently, we are actually quite uninterested in ${\displaystyle q(x)}$ itself, but rather, ${\displaystyle \nabla _{x}\ln q(x)}$. This has two major effects:

• One, we no longer need to normalize ${\displaystyle q(x)}$, but can use any ${\displaystyle {\tilde {q}}(x)=Cq(x)}$, where ${\displaystyle C=\int {\tilde {q}}(x)dx>0}$ is any unknown constant that is of no concern to us.
• Two, we are comparing ${\displaystyle q(x)}$ neighbors ${\displaystyle q(x+dx)}$, by ${\displaystyle {\frac {q(x)}{q(x+dx)}}=e^{-\langle \nabla _{x}\ln p,dx\rangle }}$

Let the score function be ${\displaystyle s(x):=\nabla _{x}\ln q(x)}$; then consider what we can do with ${\displaystyle s(x)}$.

As it turns out, ${\displaystyle s(x)}$ allows us to sample from ${\displaystyle q(x)}$ using thermodynamics. Specifically, if we have a potential energy function ${\displaystyle U(x)=-\ln q(x)}$, and a lot of particles in the potential well, then the distribution at thermodynamic equilibrium is the Boltzmann distribution ${\displaystyle q_{U}(x)\propto e^{-U(x)/k_{B}T}=q(x)^{1/k_{B}T}}$. At temperature ${\displaystyle k_{B}T=1}$, the Boltzmann distribution is exactly ${\displaystyle q(x)}$.

Therefore, to model ${\displaystyle q(x)}$, we may start with a particle sampled at any convenient distribution (such as the standard gaussian distribution), then simulate the motion of the particle forwards according to the Langevin equation

$${\displaystyle dx_{t}=-\nabla _{x_{t}}U(x_{t})dt+dW_{t}}$$

and the Boltzmann distribution is, by Fokker-Planck equation, the unique thermodynamic equilibrium. So no matter what distribution ${\displaystyle x_{0}}$ has, the distribution of ${\displaystyle x_{t}}$ converges in distribution to ${\displaystyle q}$ as ${\displaystyle t\to \infty }$.

Given a density ${\displaystyle q}$, we wish to learn a score function approximation ${\displaystyle f_{\theta }\approx \nabla \ln q}$. This is score matching.[9] Typically, score matching is formalized as minimizing Fisher divergence function ${\displaystyle E_{q}[\|f_{\theta }(x)-\nabla \ln q(x)\|^{2}]}$. By expanding the integral, and performing an integration by parts,

$${\displaystyle E_{q}[\|f_{\theta }(x)-\nabla \ln q(x)\|^{2}]=E_{q}[\|f_{\theta }\|^{2}+2\nabla \cdot f_{\theta }]+C}$$

giving us a loss function that can be minimized by stochastic gradient descent.

Suppose we need to model the distribution of images, and we want ${\displaystyle x_{0}\sim N(0,I)}$, a white-noise image. Now, most white-noise images do not look like real images, so ${\displaystyle q(x_{0})\approx 0}$ for large swaths of ${\displaystyle x_{0}\sim N(0,I)}$. This presents a problem for learning the score function, because if there are no samples around a certain point, then we can't learn the score function at that point. If we do not know the score function ${\displaystyle \nabla _{x_{t}}\ln q(x_{t})}$ at that point, then we cannot impose the time-evolution equation on a particle:

$${\displaystyle dx_{t}=\nabla _{x_{t}}\ln q(x_{t})dt+dW_{t}}$$

To deal with this problem, we perform annealing. If ${\displaystyle q}$ is too different from a white-noise distribution, then progressively add noise until it is indistinguishable from one. That is, we perform a forward diffusion, then learn the score function, then use the score function to perform a backward diffusion.

Consider again the forward diffusion process, but this time in continuous time:

$${\displaystyle x_{t}={\sqrt {1-\beta _{t}}}x_{t-1}+{\sqrt {\beta _{t}}}z_{t}}$$

By taking the ${\displaystyle \beta _{t}\to \beta (t)dt,{\sqrt {dt}}z_{t}\to dW_{t}}$ limit, we obtain a continuous diffusion process, in the form of a stochastic differential equation:

$${\displaystyle dx_{t}=-{\frac {1}{2}}\beta (t)x_{t}dt+{\sqrt {\beta (t)}}dW_{t}}$$

where ${\displaystyle W_{t}}$ is a Wiener process (multidimensional Brownian motion). Now, the equation is exactly a special case of the overdamped Langevin equation

$${\displaystyle dx_{t}=-{\frac {D}{k_{B}T}}(\nabla _{x}U)dt+{\sqrt {2D}}dW_{t}}$$

where ${\displaystyle D}$ is diffusion tensor, ${\displaystyle T}$ is temperature, and ${\displaystyle U}$ is potential energy field. If we substitute in ${\displaystyle D={\frac {1}{2}}\beta (t)I,k_{B}T=1,U={\frac {1}{2}}\|x\|^{2}}$, we recover the above equation. This explains why the phrase "Langevin dynamics" is sometimes used in diffusion models. Now the above equation is for the stochastic motion of a single particle. Suppose we have a cloud of particles distributed according to ${\displaystyle q}$ at time ${\displaystyle t=0}$, then after a long time, the cloud of particles would settle into the stable distribution of ${\displaystyle N(0,I)}$. Let ${\displaystyle \rho _{t}}$ be the density of the cloud of particles at time ${\displaystyle t}$, then we have

$${\displaystyle \rho _{0}=q;\quad \rho _{T}\approx N(0,I)}$$

and the goal is to somehow reverse the process, so that we can start at the end and diffuse back to the beginning. By Fokker-Planck equation, the density of the cloud evolves according to

$${\displaystyle \partial _{t}\ln \rho _{t}={\frac {1}{2}}\beta (t)\left(n+(x+\nabla \ln \rho _{t})\cdot \nabla \ln \rho _{t}+\Delta \ln \rho _{t}\right)}$$

where ${\displaystyle n}$ is the dimension of space, and ${\displaystyle \Delta }$ is the Laplace operator.

If we have solved ${\displaystyle \rho _{t}}$ for time ${\displaystyle t\in [0,T]}$, then we can exactly reverse the evolution of the cloud. Suppose we start with another cloud of particles with density ${\displaystyle \nu _{0}=\rho _{T}}$, and let the particles in the cloud evolve according to

$${\displaystyle dy_{t}={\frac {1}{2}}\beta (T-t)y_{t}dt+\beta (T-t)\underbrace {\nabla _{y_{t}}\ln \rho _{T-t}\left(y_{t}\right)} _{\text{score function }}dt+{\sqrt {\beta (T-t)}}dW_{t}}$$

then by plugging into the Fokker-Planck equation, we find that ${\displaystyle \partial _{t}\rho _{T-t}=\partial _{t}\nu _{t}}$. Thus this cloud of points is the original cloud, evolving backwards.[10]

At the continuous limit,

$${\displaystyle {\bar {\alpha }}_{t}=(1-\beta _{1})\cdots (1-\beta _{t})=e^{\sum _{i}\ln(1-\beta _{i})}\to e^{-\int _{0}^{t}\beta (t)dt}}$$

and so

$${\displaystyle x_{t}|x_{0}\sim N\left(e^{-{\frac {1}{2}}\int _{0}^{t}\beta (t)dt}x_{0},\left(1-e^{-\int _{0}^{t}\beta (t)dt}\right)I\right)}$$

In particular, we see that we can directly sample from any point in the continuous diffusion process without going through the intermediate steps, by first sampling ${\displaystyle x_{0}\sim q,z\sim N(0,I)}$, then get ${\displaystyle x_{t}=e^{-{\frac {1}{2}}\int _{0}^{t}\beta (t)dt}x_{0}+\left(1-e^{-\int _{0}^{t}\beta (t)dt}\right)z}$. That is, we can quickly sample ${\displaystyle x_{t}\sim \rho _{t}}$ for any ${\displaystyle t\geq 0}$. Now, define a certain probability distribution ${\displaystyle \gamma }$ over ${\displaystyle [0,\infty )}$, then the score-matching loss function is defined as the expected Fisher divergence:

$${\displaystyle L(\theta )=E_{t\sim \gamma ,x_{t}\sim \rho _{t}}[\|f_{\theta }(x_{t},t)\|^{2}+2\nabla \cdot f_{\theta }(x_{t},t)]}$$

After training, ${\displaystyle f_{\theta }(x_{t},t)\approx \nabla \ln \rho _{t}}$, so we can perform the backwards diffusion process by first sampling ${\displaystyle x_{T}\sim N(0,I)}$, then integrating the SDE from ${\displaystyle t=T}$ to ${\displaystyle t=0}$:

$${\displaystyle x_{t-dt}=x_{t}+{\frac {1}{2}}\beta (t)x_{t}dt+\beta (t)f_{\theta }(x_{t},t)dt+{\sqrt {\beta (t)}}dW_{t}}$$

This may be done by any SDE integration method, such as Euler–Maruyama method.

The name "noise conditional score network" is explained thus:

• "network", because ${\displaystyle f_{\theta }}$ is implemented as a neural network.
• "score", because the output of the network is interpreted as approximating the score function ${\displaystyle \nabla \ln \rho _{t}}$.
• "noise conditional", because ${\displaystyle \rho _{t}}$ is equal to ${\displaystyle \rho _{0}}$ blurred by an added gaussian noise that increases with time, and so the score function depends on the amount of noise added.

The original DDPM method for generating images is slow, since the forward diffusion process usually takes ${\displaystyle T\sim 1000}$ to make the distribution of ${\displaystyle x_{T}}$ to appear close to gaussian. However this means the backward diffusion process also take 1000 steps. Unlike the forward diffusion process, which can skip steps as ${\displaystyle x_{t}|x_{0}}$ is gaussian for all ${\displaystyle t\geq 1}$, the backward diffusion process does not allow skipping steps. For example, to sample ${\displaystyle x_{t-2}|x_{t-1}\sim N(\mu _{\theta }(x_{t-1},t-1),\Sigma _{\theta }(x_{t-1},t-1))}$ requires the model to first sample ${\displaystyle x_{t-1}}$. Attempting to directly sample ${\displaystyle x_{t-2}|x_{t}}$ would require us to marginalize out ${\displaystyle x_{t-1}}$, which is generally intractable.

DDIM[12] is a method to take any model trained on DDPM loss, and use it to sample with some steps skipped, sacrificing an adjustable amount of quality. The original DDPM is a special case of DDIM.

Since the diffusion model is a general method for modelling probability distributions, if one wants to model a distribution over images, one can first encode the images into a lower-dimensional space by an encoder, then use a diffusion model to model the distribution over encoded images. Then to generate an image, one can sample from the diffusion model, then use a decoder to decode it into an image.[13]

The encoder-decoder pair is most often a variational autoencoder (VAE).

Suppose we wish to sample not from the entire distribution of images, but conditional on the image description. We don't want to sample a generic image, but an image that fits the description "black cat with red eyes". Generally, we want to sample from the distribution ${\displaystyle p(x|y)}$, where ${\displaystyle x}$ ranges over images, and ${\displaystyle y}$ ranges over classes of images (a description "black cat with red eyes" is just a very detailed class, and a class "cat" is just a very vague description).

Taking the perspective of the noisy channel model, we can understand the process as follows: To generate an image ${\displaystyle x}$ conditional on description ${\displaystyle y}$, we imagine that the requester really had in mind an image ${\displaystyle x}$, but the image is passed through a noisy channel and came out garbled, as ${\displaystyle y}$. Image generation is then nothing but inferring which ${\displaystyle x}$ the requester had in mind.

In other words, conditional image generation is simply "translating from a textual language into a pictorial language". Then, as in noisy-channel model, we use Bayes theorem to get

$${\displaystyle p(x|y)\propto p(y|x)p(x)}$$

in other words, if we have a good model of the space of all images, and a good image-to-class translator, we get a class-to-image translator "for free". In the equation for backward diffusion, the score ${\displaystyle \nabla \ln p(x)}$ can be replaced by

$${\displaystyle \nabla _{x}\ln p(x|y)=\nabla _{x}\ln p(y|x)+\nabla _{x}\ln p(x)}$$

where ${\displaystyle \nabla _{x}\ln p(x)}$ is the score function, trained as previously described, and ${\displaystyle \nabla _{x}\ln p(y|x)}$ is found by using a differentiable image classifier.

The classifier-guided diffusion model samples from ${\displaystyle p(x|y)}$, which is concentrated around the maximum a posteriori estimate ${\displaystyle \arg \max _{x}p(x|y)}$. If we want to force the model to move towards the maximum likelihood estimate ${\displaystyle \arg \max _{x}p(y|x)}$, we can use

$${\displaystyle p_{\beta }(x|y)\propto p(y|x)^{\beta }p(x)}$$

where ${\displaystyle \beta >0}$ is interpretable as inverse temperature. In the context of diffusion models, it is usually called the guidance scale. A high ${\displaystyle \beta }$ would force the model to sample from a distribution concentrated around ${\displaystyle \arg \max _{x}p(y|x)}$. This often improves quality of generated images.[14] This can be done simply by SGLD with

$${\displaystyle \nabla _{x}\ln p_{\beta }(x|y)=\beta \nabla _{x}\ln p(y|x)+\nabla _{x}\ln p(x)}$$

If we do not have a classifier ${\displaystyle p(y|x)}$, we could still extract one out of the image model itself:[15]

$${\displaystyle \nabla _{x}\ln p_{\beta }(x|y)=(1-\beta )\nabla _{x}\ln p(x)+\beta \nabla _{x}\ln p(x|y)}$$

Such a model is usually trained by presenting it with both ${\displaystyle (x,y)}$ and ${\displaystyle (x,None)}$, allowing it to model both ${\displaystyle \nabla _{x}\ln p(x|y)}$ and ${\displaystyle \nabla _{x}\ln p(x)}$.

Given a diffusion model, one may regard it either as a continuous process, and sample from it by integrating a SDE, or one can regard it as a discrete process, and sample from it by iterating the discrete steps. The choice of the "noise schedule" ${\displaystyle \beta _{t}}$ can also affect the quality of samples. In the DDPM perspective, one can use the DDPM itself (with noise), or DDIM (with adjustable amount of noise). The case where one adds noise is sometimes called ancestral sampling.[16] One can interpolate between noise and no noise. The amount of noise is denoted ${\displaystyle \eta }$ ("eta value") in the DDIM paper, with ${\displaystyle \eta =0}$ denoting no noise (as in deterministic DDIM), and ${\displaystyle \eta =1}$ denoting full noise (as in DDPM).

In the perspective of SDE, one can use any of the numerical integration methods, such as Euler–Maruyama method, Heun's method, linear multistep methods, etc. Just as in the discrete case, one can add an adjustable amount of noise during the integration.

A survey and comparison of samplers in the context of image generation is in.[17]

For generating images by DDPM, we need a neural network that takes a time ${\displaystyle t}$ and a noisy image ${\displaystyle x_{t}}$, and predicts a noise ${\displaystyle \epsilon _{\theta }(x_{t},t)}$ from it. Since predicting the noise is the same as predicting the denoised image, then subtracting it from ${\displaystyle x_{t}}$, denoising architectures tend to work well. For example, the most common architecture is U-Net, which is also good at denoising images.[18]

For non-image data, we can use other architectures. For example,[19] models human motion trajectory by DDPM. Each human motion trajectory is a sequence of poses, represented by either joint rotations or positions. It uses a Transformer network to generate a less noisy trajectory out of a noisy one.

The base diffusion model can only generate unconditionally from the whole distribution. For example, a diffusion model learned on ImageNet would generate images that look like a random image from ImageNet. To generate images from just one category, one would need to impose the condition. Whatever condition one wants to impose, one needs to first convert the conditioning into a vector of floating point numbers, then feed it into the underlying diffusion model neural network. However, one has freedom in choosing how to convert the conditioning into a vector.

Stable Diffusion, for example, imposes conditioning in the form of cross-attention mechanism, where the query is an intermediate representation of the image in the U-Net, and both key and value are the conditioning vectors.[20] The conditioning can be selectively applied to only parts of an image, and new kinds of conditionings can be finetuned upon the base model, as used in ControlNet.[20]

As a particularly simple example, consider image inpainting. The conditions are ${\displaystyle {\tilde {x}}}$, the reference image, and ${\displaystyle m}$, the inpainting mask. The conditioning is imposed at each step of the backward diffusion process, by first sampling ${\displaystyle {\tilde {x}}_{t}\sim N\left({\sqrt {{\bar {\alpha }}_{t}}}{\tilde {x}},(1-{\bar {\alpha }}_{t})I\right)}$, a noisy version of ${\displaystyle {\tilde {x}}}$, then replacing ${\displaystyle x_{t}}$ with ${\displaystyle (1-m)\odot x_{t}+m\odot {\tilde {x}}_{t}}$, where ${\displaystyle \odot }$ means elementwise multiplication.[21]

Conditioning is not limited to just generating images from a specific category, or according to a specific caption (as in text-to-image). For example,[19] demonstrated generating human motion, conditioned on an audio clip of human walking (allowing syncing motion to a soundtrack), or video of human running, or a text description of human motion, etc.

As generating an image takes a long time, one can try to generate a small image by a base diffusion model, then upscale it by other models. Upscaling can be done by GAN,[22] Transformer,[23] or signal processing methods like Lanczos resampling.

Diffusion models themselves can be used to perform upscaling. Cascading diffusion model stacks multiple diffusion models one after another, in the style of Progressive GAN. The lowest level is a standard diffusion model that generate 32x32 image, then the image would be upscaled by a diffusion model specifically trained for upscaling, and the process repeats.[18]

The DALL-E series by OpenAI are text-conditional diffusion models of images.

The first version of DALL-E (2021) is not actually a diffusion model. Instead, it uses a Transformer architecture that generates a sequence of tokens, which is then converted to an image by the decoder of a discrete VAE. Released with DALL-E was the CLIP classifier, which was used by DALL-E to rank generated images according to how close the image fits the text.

GLIDE (2022-03)[24] is a 3.5-billion diffusion model, and a small version was released publicly.[25] Soon after, DALL-E 2 was released (2022-04).[26] DALL-E 2 is a 3.5-billion cascaded diffusion model that generates images from text by "inverting the CLIP image encoder", the technique which they termed "unCLIP".

Stable Diffusion (2022-08), released by Stability AI, consists of a latent diffusion model (860 million parameters), a VAE, and a text encoder. The diffusion model is a U-Net, with cross-attention blocks to allow for conditional image generation.[27][13]

Google Imagen[28] and Imagen Video[29] are two cascaded diffusion models for generating images and videos.[30] They use a T5-XXL, a Transformer-based language model, to encode text for text-conditional generation.

Make-a-video by Meta AI[31] generates videos from text.

DreamFusion[32][33] generates 3D models from text.