The Generative Modelling Problem

By 2014, deep networks could classify images and recognise speech remarkably well. Asking the same networks to create a convincing image was a different story. This lesson explains what a generative model is, why the popular approaches of the time were so hard to train, and the one idea that made GANs different.

Two Kinds of Model

A discriminative model learns a conditional distribution p(y|x): given an input x (an image), how likely is each label y ("cat", "dog")? It only has to find the boundary between classes. It never needs to know what a typical cat image looks like, only which side of the boundary it falls on.

A generative model learns the distribution of the data itself, p(x). It has to capture everything: which pixel patterns are common, which are rare, and which never happen. If you have p(x), or at least a way to sample from it, you can produce brand new examples that look like they came from the training set.

The generative task is much harder. A 64×64 colour image has 12,288 numbers, and real images occupy a tiny, intricately shaped region of that space. The paper opens with exactly this contrast: the striking successes of deep learning had involved discriminative models, while deep generative models "have had less of an impact, due to the difficulty of approximating many intractable probabilistic computations that arise in maximum likelihood estimation".

The 2014 Landscape

The standard recipe was maximum likelihood: write down a model density p_model(x; θ) and adjust θ to make the training data as probable as possible. That requires p_model to be a proper probability distribution, which is where the trouble starts.

Restricted and deep Boltzmann machines

An RBM assigns every configuration an energy and defines p(x) = exp(−E(x)) / Z. The normaliser Z, the partition function, is a sum over every possible configuration of the units. For binary units that is 2^n terms, and the paper notes that the partition function and its gradient are intractable for all but the most trivial instances. They can be estimated with Markov chain Monte Carlo (MCMC), but MCMC chains mix slowly between modes. Deep Boltzmann machines stack more layers and inherit the same problem. Deep belief networks are a hybrid of directed and undirected layers and still carry an intractable undirected part.

Other routes

Score matching and noise-contrastive estimation avoid computing Z directly, but still need the density known up to a normaliser. Generative stochastic networks learn a Markov chain whose samples come from the model, so sampling still means running a chain. In the same year, variational autoencoders (Kingma and Welling; Rezende et al.) paired a generator network with an inference network and maximised a lower bound on the log-likelihood. VAEs still need an explicit likelihood p(x|z) for that bound.

The GAN Idea: Learn a Sampler, Not a Density

GANs ask a different question. Instead of "what is p(x)?", they ask "can I build a machine whose outputs are indistinguishable from real data?". The machine is a neural network G that turns random noise into a sample. Nobody ever writes down the density of its outputs. To judge G, a second network D is trained to tell real samples from generated ones, and G is trained to fool D.

The paper puts it with an analogy:

The generative model can be thought of as analogous to a team of counterfeiters, trying to produce fake currency and use it without detection, while the discriminative model is analogous to the police, trying to detect the counterfeit currency. Competition in this game drives both teams to improve their methods until the counterfeits are indistinguishable from the genuine articles.

When both players are multilayer perceptrons, the whole system trains with ordinary backpropagation, and sampling is a single forward pass. In the paper's words, there is no need for any Markov chains or unrolled approximate inference networks during either training or generation of samples.