Identity Shortcut Connections
y = F(x, {Wᵢ}) + x, free of parameters, and what to do when shapes change
By the end of this lesson you can
- Read Eq. 1 and Eq. 2 and say where the ReLU goes
- Explain why parameter-free shortcuts make the plain vs residual comparison fair
- Distinguish shortcut options A, B and C and implement A and B on feature maps
Residual learning needs a way to carry x past the layers and add it back. The paper does this with the simplest possible wiring: a shortcut that skips the layers and an element-wise addition. This lesson covers the equation, why it costs nothing, and what to do when the shapes on the two sides do not match.
Equation 1: The Building Block
Section 3.2 defines a building block as:
Here x and y are the input and output vectors of the layers considered, and F(x, {Wᵢ}) is the residual mapping to be learned. For the two-layer example of Figure 2 (biases omitted to simplify notation):
output = σ(y) = σ(F + x) (the second nonlinearity comes after the addition)
The operation F + x is performed by a shortcut connection and element-wise addition. For feature maps it is channel by channel: the value at each channel and pixel of F is added to the same channel and pixel of x.
Free of Parameters
An identity shortcut has no weights and costs one addition per element, negligible next to the convolutions it skips. The paper stresses why this matters beyond efficiency: it allows a fair comparison. A plain network and its residual counterpart have the same number of parameters, the same depth, width and computational cost. Any difference in accuracy comes from the reformulation, not from extra capacity.
The residual function F is flexible: the paper uses two or three layers, and more are possible. But a single layer does not help. With one linear layer, Eq. 1 becomes y = W₁x + x = (W₁ + I)x, which is just another linear layer, and the authors report they have not observed advantages for it.
Equation 2: When Dimensions Change
F(x) and x must have the same shape to be added. Inside a stage they do. At the start of a new stage the network halves the spatial size and doubles the number of channels, so F(x) no longer matches x. The paper's general fix is a linear projection W_s on the shortcut:
W_s could be used everywhere, even as a square matrix. The paper shows (Table 3, lesson 11) that the identity is sufficient for addressing degradation and is economical, so W_s is only used when matching dimensions. On feature maps, W_s is a 1×1 convolution.
Three Options for the Shortcut
Section 4.1 compares three ways to handle the shortcuts:
For both A and B, when the shortcut goes across feature maps of two sizes it is performed with a stride of 2. For option A that means taking every second pixel and appending zero channels: a 64-channel, 56×56 input becomes a 128-channel, 28×28 tensor whose last 64 channels are all zero. For option B the 1×1 convolution itself has stride 2.
Shortcuts Before ResNet
Shortcut connections were not new. Early MLPs added a linear layer from input to output, GoogLeNet and deeply supervised nets attached auxiliary classifiers to intermediate layers, and, closest of all, highway networks (Srivastava et al., 2015) used shortcuts with gating functions. Highway gates are data-dependent and have parameters. When a gate closes, the layer becomes non-residual. ResNet's identity shortcuts are never closed: all information always passes through, and there is always a residual function to learn. The paper also notes that highway networks had not demonstrated accuracy gains at extreme depths (over 100 layers).
Edit the code and press Ctrl + Enter to run it in your browser.
How expensive are projection shortcuts relative to the whole network? Count the main-path weights of ResNet-34 (conv1, all 3×3 convs, and the fc layer) and express the option B and option C shortcut weights as a percentage of it.
Key takeaway
An identity shortcut plus element-wise addition costs no parameters and almost no compute. When dimensions change, the shortcut either zero-pads (option A) or uses a strided 1×1 projection W_s (option B).
Up next
Before building full blocks, we need fluency in the 3×3 convolutions they are made of.