Chapter 1 — The role of forward diffusion

In the Fundamentals tutorial, we viewed the data generation problem through the lens of a ball exploring a landscape in search of high peaks. To overcome the sparse nature of the landscape (which caused the ball to get stuck in flat regions), we dynamically changed the shape of the $p(x)$ terrain over time. The new challenge is modeling a time-variant probability density $p_t(x)$—or rather, the more tractable score function $\nabla_x \log p_t(x)$.

So how can we learn it from a training set consisting of unrelated images haphazardly scraped from the Internet? Afterall, they are nothing more than some isolated static points in our landscape.

The core insight is that, while individual diffusing particles move randomly, a cloud of particles evolves deterministically. Just like a drop of ink diffusing in water will predictably stabilize to a state where ink and water particles are mixed homogeneously.

This is one of the useful discoveries that physics gives us for free. If particles are distributed as $p_t(x)$ at time $t$, the Fokker-Planck equation gives a deterministic expression for $\partial_t p_t(x)$, which describes how it evolves over time. You can find the exact formula on Wikipedia, but what matters most is that such a deterministic formula exists.

The implication is that, when we simulate individual particles, their empirical distribution $p_t(x)$ approximates the quantity we aim to model. Hence, simulating diffusing particles can give us valuable training signal.

In the rest of this tutorial, we derive a forward-diffusion equation step by step and turn it into a practical particle simulator.