Chapter 1 — From data to supervision

In Forward diffusion, we derived a formula for particle motion from clean data to pure noise. Since this process is known, we can apply it programmatically to corrupt real images and generate supervision for training.

In practice, we discretize a continuous sample path $x$ into a chain of progressively noisier states:

$$x_0 \rightarrow x_1 \rightarrow \dots x_{T-1}$$

where $x_0$ is a real image from the dataset and the final state $x_{T-1}$ is pure noise.

You might expect that we should explicitly generate the full chain for each $x_0$. But in practice, each iteration usually samples a single random time $t$. The model then sees one pair: for example $(x_t \to x_{t-1})$ or $(x_t \to x_0)$, depending on the exact training objective.

But why not train on the entire chain? Isn't more training data always better?

In practice, training has a finite budget of $N$ iterations. Spending many consecutive updates on one chain from a fixed $x_0$ gives highly correlated examples. Sampling fresh $(x_0, t)$ pairs each iteration typically covers the space more efficiently under the same budget. With enough compute, you would still eventually visit the full chain anyway.

To express $x_t$, don't we need to express all $x_{<t}$? If so, isn't it a waste to not train on $x_{<t}$?

For DDPM-style forward processes, we can sample $x_t$ directly from $x_0$ and $t$ via a closed-form distribution, so we do not need to explicitly generate all $x_{<t}$ first.

The overall training data generation algorithm

1. Draw $x_0$ from the training set.
2. Draw a random time step $t$.
3. Corrupt $x_0$ into $x_t$ using the forward motion.
4. Train the model to predict the denoising target associated with $x_t$ (more on objectives soon).