From Noise to Data
The generator as a function that reshapes a simple distribution
By the end of this lesson you can
- Understand the prior p_z(z) and the generator G(z; θ_g)
- See how a deterministic function of random noise defines a new distribution p_g
- Relate slopes of G to where probability mass piles up
A generator is surprisingly simple: take a random number you know how to draw, feed it through a function, and call the output a sample. All the richness of the generated distribution comes from the shape of that function.
The Setup in the Paper
Section 3 of the paper defines the generator in one sentence: to learn the generator's distribution p_g over data x, define a prior on input noise variables p_z(z), then represent a mapping to data space as G(z; θ_g), where G is a differentiable function represented by a multilayer perceptron with parameters θ_g.
x = G(z; θ_g) (a deterministic, differentiable function)
⇒ x ∼ p_g (the distribution induced by G)
Three objects, three roles. The prior p_z is fixed and easy to sample. The generator G is the only thing that learns. The model distribution p_g is never written down: it is whatever you get by pushing p_z through G. Training changes θ_g, which reshapes p_g.
A Deterministic Function Makes a New Distribution
G itself contains no randomness. All the randomness comes from z. Yet because z is random, G(z) is random too, and its distribution depends on G. You already know simple cases:
- If z ∼ N(0, 1) and G(z) = μ + σz, then p_g = N(μ, σ²). A linear G can only shift and stretch.
- If z ∼ Uniform(0, 1) and G is the inverse CDF of any distribution, then p_g is exactly that distribution. This is inverse transform sampling, and it shows a smooth enough function can turn uniform noise into anything in one dimension.
An MLP is a flexible, nonlinear function, so it can bend, fold and stretch the noise into complicated shapes. That is why the paper uses neural networks for G.
Where Mass Piles Up: Change of Variables
In one dimension, if G is increasing and invertible, the change-of-variables formula tells us the density of x = G(z):
Read it as bookkeeping. A small interval of width dz around z holds probability p_z(z)·dz. G maps it to an interval of width |G′(z)|·dz. The same probability squeezed into a narrower interval means higher density:
- Where G is flat (small slope), many z values map to a small range of x, so probability piles up and p_g has a peak.
- Where G is steep, nearby z values are spread far apart in x, so the density thins out.
The paper's Figure 1 draws exactly this: arrows from uniformly spaced z values to their images G(z), bunched together where p_g is high. Its caption says G contracts in regions of high density and expands in regions of low density of p_g.
Why No Density Is Needed
In real GANs z might have 100 dimensions while x has thousands, and G is not invertible. The generated samples then lie on a low-dimensional surface inside data space, and p_g may not even have a density in the usual sense. That does not matter: training only ever needs samples G(z) and gradients through G. This is the freedom bought by learning a sampler instead of a density.
Edit the code and press Ctrl + Enter to run it in your browser.
Let z ∼ Uniform(0, 1) and G(z) = z². Derive p_g(x) with the change-of-variables formula. Where does the mass pile up, and why?
Your generator is linear, G(z) = a·z + b, with z ∼ N(0, 1). Which data distributions can it represent exactly? What does this tell you about why the paper uses a multilayer perceptron?
In a real GAN, z might be 100-dimensional while x is a 64×64×3 image. Does p_g have a density that you could compute with a change-of-variables formula? Does training need one?
Key takeaway
Push simple noise through a learned function and you get samples from a new distribution p_g, with no density formula required.
Up next
But how do we know if p_g looks like the data? We train a critic.