The Degradation Problem

By 2015 everyone agreed that depth was the secret ingredient of deep learning. VGG had 19 layers, GoogLeNet 22, and each jump in depth had bought a jump in ImageNet accuracy. So why not 50 layers, or 100? The ResNet paper opens with an uncomfortable experiment: simply stacking more layers makes a network worse, and not in the way anyone expected.

Is Learning Better Networks as Easy as Stacking More Layers?

That is almost word for word the question the paper asks. The obvious obstacle, vanishing or exploding gradients, had largely been handled by then. Careful initialisation (the “normalized initialization” of Glorot and of He et al.) and intermediate normalisation layers (batch normalization) let networks with tens of layers start converging under plain stochastic gradient descent with backpropagation.

Yet once deeper networks could start converging, a second problem appeared. In the paper's words: with the network depth increasing, accuracy gets saturated (which might be unsurprising) and then degrades rapidly. The authors call this the degradation problem.

Figure 1: More Layers, Higher Training Error

Figure 1 trains two “plain” networks on CIFAR-10, one with 20 layers and one with 56. Plain means an ordinary stack of convolutions, BN and ReLU, with no shortcut connections. The 56-layer network ends with higher error than the 20-layer one, and crucially this holds for the training error as well as the test error. The same happens on ImageNet: the 34-layer plain net is worse than the 18-layer plain net (Table 2).

The first column is the problem of this lesson: going from 18 to 34 plain layers makes things worse. The second column is the answer the rest of the course builds: with shortcut connections, the extra depth pays off.

Why It Is Not Overfitting

Overfitting has a clear signature: a bigger model fits the training set better but generalises worse. The deeper plain net does the opposite. It fits the training set worse, from early in training to the end (Figure 4, left, shows the 34-layer training curve above the 18-layer one throughout). A model that cannot even match a smaller model on the data it is trained on is not memorising too much. It is failing to optimise.

The Construction Argument

Here is the argument that makes the result genuinely surprising. Take a trained shallow network. Build a deeper one by copying its layers and inserting extra layers that compute the identity. This deeper network computes exactly the same function, so its training error is exactly the same.

So a solution at least as good as the shallow model exists in the deeper model's parameter space. The paper puts it plainly: the existence of this constructed solution indicates that a deeper model should produce no higher training error than its shallower counterpart. But experiments show that current solvers are unable to find solutions that are comparably good or better, or at least not in feasible time. The failure belongs to the optimiser, not to the model family.

What the Paper Does and Does Not Blame

It is tempting to say “vanishing gradients” and move on. The paper explicitly argues against that for its own plain networks (Section 4.1). They are trained with BN, which ensures forward-propagated signals have non-zero variance, and the authors verified that backward-propagated gradients have healthy norms. The 34-layer plain net even reaches competitive accuracy, so the solver works to some extent. Instead the authors conjecture that deep plain nets may have exponentially low convergence rates, which slow the reduction of training error. The exact cause is left open for future study.

The code cell reproduces the shape of Figure 1 on a toy problem. It also builds the construction argument explicitly: a 6-layer net padded with 42 identity layers matches the 6-layer loss, while a 48-layer net trained from scratch does far worse. (The toy has no BN, so it is an analogy for the phenomenon, not a replica of the paper's setup.)