Learn
Lesson 04 / 13Core25 min

Convolution Arithmetic

3×3 kernels, stride, padding, output sizes, parameters and FLOPs

By the end of this lesson you can

  • Compute output sizes with floor((n + 2p - k) / s) + 1
  • Count parameters and multiply-adds of a convolutional layer
  • Explain the halve-the-map, double-the-filters rule and reproduce the 3.6 billion FLOPs of ResNet-34
Visual explainer
Loading lab
Explanation

ResNet is built almost entirely from 3×3 convolutions. Before looking at blocks and stages, it pays to be fluent in the arithmetic every architecture table assumes: how big the output is, how many weights a layer has, and how much computation it costs.

What a Convolution Computes

A convolutional layer takes an input with C_in channels and produces C_out channels. Each output channel has its own k×k×C_in filter. At every output position the filter is laid over a k×k window of the input, across all input channels, and the products are summed:

out[o, r, c] = Σ_ch Σ_i Σ_j  w[o, ch, i, j] · x[ch, s·r + i, s·c + j]

Two knobs control the geometry. The stride s is how far the window moves between outputs. The padding p adds a border of zeros around the input so windows can be centred on edge pixels.

Output Size

n_out = ⌊(n + 2p − k) / s⌋ + 1

Three cases account for almost every layer in ResNet:

  • 3×3, stride 1, padding 1: n_out = n. The map keeps its size (“same” convolution).
  • 3×3, stride 2, padding 1: n_out = ⌊(n − 1)/2⌋ + 1, which halves an even n (56 → 28).
  • 1×1, stride 1, no padding: n_out = n. A per-pixel linear map across channels.

Following the ImageNet stem of Table 1: a 224×224 image goes through conv1 (7×7, 64 filters, stride 2) to 112×112, then a 3×3 max pool with stride 2 to 56×56. The stages then run at 56, 28, 14 and 7, with the downsampling done by conv3_1, conv4_1 and conv5_1 at stride 2.

Parameters

params = k · k · C_in · C_out    (+ C_out biases, which ResNet omits because BN follows)

A 3×3 conv from 64 to 64 channels has 9 · 64 · 64 = 36,864 weights. The count does not depend on the image size: the same filters slide over every position. That weight sharing is why convolutions are so parameter-efficient.

Multiply-Adds (“FLOPs” in the Paper)

Every output value needs k·k·C_in multiply-adds, and there are C_out · H_out · W_out output values:

mult-adds = k · k · C_in · C_out · H_out · W_out = params · H_out · W_out

The paper reports complexity in FLOPs and says explicitly that it counts multiply-adds. With these two formulas you can check the architecture's key design rule.

The Plain Network Design Rules

The plain baselines (and hence ResNets) are inspired by the philosophy of VGG nets. Section 3.3 states two rules: (i) for the same output feature map size, the layers have the same number of filters; and (ii) if the feature map size is halved, the number of filters is doubled so as to preserve the time complexity per layer.

3x3 conv layer params mult-adds 64 -> 64 at 56x56 36,864 115,605,504 128 -> 128 at 28x28 147,456 115,605,504 256 -> 256 at 14x14 589,824 115,605,504 512 -> 512 at 7x7 2,359,296 115,605,504

Halving H and W divides the positions by 4; doubling C_in and C_out multiplies the weights by 4. The cost per layer is constant, while parameters grow toward the deep end of the network. Stacking 3×3 layers also grows the receptive field cheaply: two 3×3 layers see a 5×5 window with 18C² weights instead of 25C², plus an extra nonlinearity.

Summed over the whole 34-layer network, this comes to 3.6 billion FLOPs, only 18% of VGG-19's 19.6 billion. The paper notes that its models have fewer filters and lower complexity than VGG nets. The first exercise reproduces the 3.6 billion from the formulas above.

Common mistake: assuming parameters and compute scale together. A 512-channel 3×3 layer at 7×7 has 64 times the weights of a 64-channel layer at 56×56, yet exactly the same multiply-adds. Parameters are set by channels and kernel size; compute also depends on the spatial size of the output.
Interactive code

Edit the code and press Ctrl + Enter to run it in your browser.

pynumpy · matplotlib
Exercises
1

The paper says ResNet-34 costs 3.6 billion FLOPs (multiply-adds), 18% of VGG-19's 19.6 billion. Reproduce the number: sum the multiply-adds of conv1, every 3×3 conv in the four stages (remember where the map is halved), and the fc layer.

2

Why stack small kernels? Compare the weights of two stacked 3×3 convs with one 5×5 conv at 64 channels, and verify with conv2d that two 3×3 layers see a 5×5 window of the input.

Key takeaway

Parameters are k·k·C_in·C_out; multiply-adds multiply that by the output area. Halving the map while doubling the filters keeps the cost per layer constant, which is how ResNet-34 stays at 3.6 billion FLOPs, 18% of VGG-19.

Up next

Every one of those convolutions is followed by batch normalization, the topic of the next lesson.