Learn
Lesson 02 / 13Warm-up20 min

Residual Learning

Learn F(x) = H(x) - x instead of H(x)

By the end of this lesson you can

  • Write the residual reformulation F(x) := H(x) - x and the block output F(x) + x
  • Explain why the identity is easy for a residual block and hard for a plain one
  • Frame residual learning as preconditioning, and name the evidence from Figure 7
Visual explainer
Loading lab
Explanation

If identity layers are what the solver fails to build, make the identity free. That one-line idea is the whole of residual learning: instead of asking a stack of layers for the function you want, ask it for the difference between that function and its input.

Reformulating the Target

Let H(x) be the mapping that a few stacked layers ought to compute, with x the input to the first of them. A plain network asks the layers to fit H(x) directly. Section 3.1 of the paper proposes instead to let the layers fit a residual function:

F(x) := H(x) − x    so the block outputs    F(x) + x

The original mapping is recast as F(x) + x. Nothing about expressive power has changed. The paper states the hypothesis carefully: if multiple nonlinear layers can asymptotically approximate complicated functions, they can equally approximate the residual H(x) − x (assuming input and output have the same dimensions). Both forms can represent the desired function. What differs, the authors argue, is the ease of learning.

Why the Identity Becomes Easy

Return to the degradation problem. It suggests that solvers have difficulty approximating identity mappings with several nonlinear layers. Compare the two ways a block can represent “do nothing”:

plain:      W₂ σ(W₁ x) = x    (a precise, coordinated setting of all weights)
residual: W₂ σ(W₁ x) + x = x  ⇐  W₂ = 0    (drive the weights toward zero)

With a ReLU in the middle, a plain block has to reconstruct negative values from non-negative activations, for example x = relu(x) − relu(−x), and get every weight exactly right. The residual block gets the identity by doing nothing at all. In the paper's words, if identity mappings are optimal, the solvers may simply drive the weights of the multiple nonlinear layers toward zero to approach identity mappings. Weight decay and small initialisations already push in that direction.

Preconditioning, Not a Guarantee

The paper does not claim that identity mappings are usually optimal. It says the opposite: in real cases, it is unlikely that identity mappings are optimal. The claim is weaker and more useful. If the optimal function is closer to an identity mapping than to a zero mapping, it should be easier for the solver to find the perturbations with reference to an identity than to learn the function as a new one. The reformulation acts as a preconditioner: it moves the default answer of every block from “zero” to “pass the input through”.

Is the premise true? The paper checks it empirically in Figure 7 (lesson 12): on CIFAR-10, the standard deviations of layer responses in ResNets are generally smaller than in the corresponding plain networks, and deeper ResNets have smaller responses still. The learned residual functions really are close to zero, so the identity is a good reference point.

An Old Idea in a New Place

Section 2 connects residual learning to older work. In image retrieval and classification, VLAD and Fisher vectors encode residual vectors with respect to a dictionary, and encoding residuals was known to work better than encoding the original vectors. In numerical methods for partial differential equations, multigrid methods solve for residuals between coarse and fine scales, and hierarchical basis preconditioning reformulates variables as residuals. These solvers converge much faster than ones unaware of the residual structure. The common thread: a good reformulation can make an optimisation problem easier without changing its solution.

The key insight to carry forward: a residual block does not change what the network can represent. It changes the starting point. Every block defaults to the identity, so a very deep ResNet starts out behaving like a much shallower one, and each block only has to learn a small correction.
Interactive code

Edit the code and press Ctrl + Enter to run it in your browser.

pynumpy · matplotlib
Exercises
1

Residual learning is a bet that the target is close to the identity. Replace the target with one that is not, Y = tanh(2 X B) for a random orthogonal B, and rerun both fits. What happens to the advantage of the residual form?

2

Inspect what the residual branch actually learned. Compare the standard deviation of the learned F(x) with that of the input x and of the true residual H(x) - x.

Key takeaway

A residual block learns a correction to its input rather than a whole new function. Expressive power is unchanged; what changes is the default, which becomes the identity, and that helps whenever the optimal function is closer to the identity than to zero.

Up next

Next: the wiring that carries x around the layers, and what happens when shapes change.