Learn
Lesson 05 / 13Core20 min

Batch Normalization in ResNet

Conv, then BN, then ReLU: per-channel statistics, train vs inference

By the end of this lesson you can

  • Implement per-channel BN over (N, H, W) with learned γ and β
  • Contrast batch statistics in training with running statistics at inference
  • Explain why BN lets the paper rule out vanishing signals as the cause of degradation
Visual explainer
Loading lab
Explanation

Every convolution in ResNet is followed by batch normalization. It is easy to treat BN as plumbing, but it is part of the paper's argument: BN is why the authors can say their plain networks do not suffer from vanishing signals, and so why degradation must be something else.

Where BN Sits

Section 3.4: “We adopt batch normalization (BN) right after each convolution and before activation”, following Ioffe and Szegedy (2015). The standard unit of a ResNet is therefore:

conv → BN → ReLU

The paper uses no dropout, following the practice of the BN paper. Inside a residual block, the second BN comes before the addition and the last ReLU after it (lesson 6).

Per-Channel Statistics

For a fully connected layer, BN normalises each feature over the mini-batch. For a convolutional layer the statistics are shared across space as well, to respect the fact that one filter is applied at every position. With activations of shape (N, C, H, W), each channel c gets one mean and one variance, computed over N · H · W numbers:

μ_c = mean over (n, h, w) of x[n, c, h, w]
σ²_c = mean over (n, h, w) of (x[n, c, h, w] − μ_c)²
x̂ = (x − μ_c) / √(σ²_c + ε)
y = γ_c · x̂ + β_c

After normalisation every channel has zero mean and unit variance over the batch. The learned scale γ_c and shift β_c then let the network choose whatever mean and spread it wants, including undoing the normalisation entirely. Each BN layer adds 2C parameters.

Train Mode vs Inference Mode

During training, μ and σ² come from the current mini-batch, so an image's output depends on which other images share its batch. That is fine during training (the BN paper even reports a mild regularising effect) but unacceptable at test time, where a prediction should depend only on the input. So BN keeps running averages of the statistics during training and uses them, frozen, at inference:

running_mean ← (1 − m) · running_mean + m · μ_batch
running_var  ← (1 − m) · running_var  + m · σ²_batch    (m = 0.1 in PyTorch)

Implementations differ in small details: PyTorch normalises with the biased batch variance but stores the unbiased one in the running average. Forgetting to switch a model to inference mode is one of the most common deployment bugs.

Why a Conv Bias Is Redundant

A bias added by the convolution is a per-channel constant, and BN immediately subtracts the per-channel mean. The bias cancels exactly. Its job is taken over by β, so ResNet convolutions have no bias. At inference, BN is just a fixed per-channel affine map, which can be folded into the preceding convolution's weights (exercise 1).

BN and the Degradation Argument

BN is what lets the paper separate degradation from vanishing signals. Its plain networks are trained with BN, which ensures forward-propagated signals have non-zero variances, and the authors verified that backward-propagated gradients exhibit healthy norms with BN. Neither forward nor backward signals vanish, and the 34-layer plain net still trains worse than the 18-layer one. BN makes deep plain nets trainable; it does not make them easy to optimise. BN also provides the measurement behind Figure 7: the layer responses the paper analyses are the outputs of each 3×3 layer after BN and before the nonlinearity or addition.

Common mistake: thinking BN normalises each pixel or each image separately. In a conv net, BN computes one mean and one variance per channel, pooled over the batch and all spatial positions, and at test time it uses the stored running statistics, not the test batch.
Interactive code

Edit the code and press Ctrl + Enter to run it in your browser.

pynumpy · matplotlib
Exercises
1

At inference, BN is a fixed per-channel affine map, so it can be folded into the conv before it. Derive the folded weights and bias, then check numerically that conv followed by BN equals the folded conv.

2

Batch statistics are estimates. For batch sizes 1, 4, 32 and 256, draw 200 batches and measure how much the batch mean of channel 3 (true mean 10, std 5) fluctuates.

Key takeaway

ResNet puts BN right after every convolution and before the activation. BN normalises each channel with batch statistics during training and stored running averages at inference, makes conv biases redundant, and keeps forward and backward signals healthy, which is why degradation must be an optimisation problem.

Up next

Now we can assemble conv, BN, ReLU and the shortcut into the basic block.