Learn
Lesson 02 / 14Warm-up15 min

From Noise to Data

The generator as a function that reshapes a simple distribution

By the end of this lesson you can

  • Understand the prior p_z(z) and the generator G(z; θ_g)
  • See how a deterministic function of random noise defines a new distribution p_g
  • Relate slopes of G to where probability mass piles up
Visual explainer
Loading lab
Explanation

A generator is surprisingly simple: take a random number you know how to draw, feed it through a function, and call the output a sample. All the richness of the generated distribution comes from the shape of that function.

The Setup in the Paper

Section 3 of the paper defines the generator in one sentence: to learn the generator's distribution p_g over data x, define a prior on input noise variables p_z(z), then represent a mapping to data space as G(z; θ_g), where G is a differentiable function represented by a multilayer perceptron with parameters θ_g.

z ∼ p_z(z)    (simple noise, e.g. uniform or Gaussian)
x = G(z; θ_g)    (a deterministic, differentiable function)
⇒ x ∼ p_g    (the distribution induced by G)

Three objects, three roles. The prior p_z is fixed and easy to sample. The generator G is the only thing that learns. The model distribution p_g is never written down: it is whatever you get by pushing p_z through G. Training changes θ_g, which reshapes p_g.

A Deterministic Function Makes a New Distribution

G itself contains no randomness. All the randomness comes from z. Yet because z is random, G(z) is random too, and its distribution depends on G. You already know simple cases:

  • If z ∼ N(0, 1) and G(z) = μ + σz, then p_g = N(μ, σ²). A linear G can only shift and stretch.
  • If z ∼ Uniform(0, 1) and G is the inverse CDF of any distribution, then p_g is exactly that distribution. This is inverse transform sampling, and it shows a smooth enough function can turn uniform noise into anything in one dimension.

An MLP is a flexible, nonlinear function, so it can bend, fold and stretch the noise into complicated shapes. That is why the paper uses neural networks for G.

Where Mass Piles Up: Change of Variables

In one dimension, if G is increasing and invertible, the change-of-variables formula tells us the density of x = G(z):

p_g(x) = p_z(z) / |G′(z)|    where z = G⁻¹(x)

Read it as bookkeeping. A small interval of width dz around z holds probability p_z(z)·dz. G maps it to an interval of width |G′(z)|·dz. The same probability squeezed into a narrower interval means higher density:

  • Where G is flat (small slope), many z values map to a small range of x, so probability piles up and p_g has a peak.
  • Where G is steep, nearby z values are spread far apart in x, so the density thins out.

The paper's Figure 1 draws exactly this: arrows from uniformly spaced z values to their images G(z), bunched together where p_g is high. Its caption says G contracts in regions of high density and expands in regions of low density of p_g.

The generator's job: learning G means learning where to be flat and where to be steep, so that the mass of p_g lands where the mass of p_data is. The formula above is only for intuition; the GAN never computes it.

Why No Density Is Needed

In real GANs z might have 100 dimensions while x has thousands, and G is not invertible. The generated samples then lie on a low-dimensional surface inside data space, and p_g may not even have a density in the usual sense. That does not matter: training only ever needs samples G(z) and gradients through G. This is the freedom bought by learning a sampler instead of a density.

Common mistake: thinking G adds noise to its output, or that sampling twice from the same z gives different results. G is deterministic: the same z always gives the same x. Variety in the samples comes only from drawing different z.
Interactive code

Edit the code and press Ctrl + Enter to run it in your browser.

pynumpy · matplotlib
Exercises
1

Let z ∼ Uniform(0, 1) and G(z) = z². Derive p_g(x) with the change-of-variables formula. Where does the mass pile up, and why?

2

Your generator is linear, G(z) = a·z + b, with z ∼ N(0, 1). Which data distributions can it represent exactly? What does this tell you about why the paper uses a multilayer perceptron?

3

In a real GAN, z might be 100-dimensional while x is a 64×64×3 image. Does p_g have a density that you could compute with a change-of-variables formula? Does training need one?

Key takeaway

Push simple noise through a learned function and you get samples from a new distribution p_g, with no density formula required.

Up next

But how do we know if p_g looks like the data? We train a critic.