Convolution operation
A convolution in the context of a neural network is a sliding-window operation where a small grid of learned weights, called a filter or kernel, is moved a...
The Mechanics of a Single Convolution
A convolution in the context of a neural network is a sliding-window operation where a small grid of learned weights, called a filter or kernel, is moved across an input array. At each position, the filter is placed over a patch of the input, corresponding values are multiplied element-wise, and the products are summed to produce a single output value. The collection of all such output values forms a two-dimensional feature map that encodes how strongly the pattern described by the filter appears at each location in the input.
For example, a 3-by-3 filter trained to detect horizontal edges will produce large positive values wherever the input image transitions from dark to bright horizontally. This local pattern detection is the core reason convolutions are so effective for images and other spatially structured data.
Exam tip: Be prepared to compute the output size of a convolution by hand. The formula is output size equals input size minus kernel size plus one, adjusted for stride and padding. Walking through a small numerical example in an interview demonstrates genuine understanding rather than memorized vocabulary.
Weight Sharing and Translation Equivariance
A key property of convolutions is that the same filter weights are applied at every spatial position in the input. This weight sharing dramatically reduces the number of parameters compared to a fully connected layer of the same input size. A fully connected layer connecting a 224-by-224 image to a hidden layer of 1000 units would require over fifty million parameters. A single 3-by-3 convolutional filter accomplishes its spatial scan with only nine parameters.
Weight sharing also produces translation equivariance: if a feature appears at the upper-left corner of the image or the lower-right corner, the same filter detects it with equal effectiveness at both locations. This property aligns with the natural structure of visual data, where the identity of an object does not depend on where it appears in the frame.
Understanding weight sharing is essential for explaining why convolutional networks generalize well with relatively modest amounts of training data compared to fully connected architectures applied to images.
Depth, Stacking Layers, and Receptive Field
A single convolutional layer detects local patterns within its kernel window, but useful visual concepts such as faces, cars, or handwritten digits involve features at multiple scales. By stacking convolutional layers, deeper layers effectively look at larger regions of the original input because each unit in a deeper layer aggregates information from a wider patch of the previous layer's output. The size of the original input region that influences a single unit in a deeper layer is called its receptive field.
With a 3-by-3 kernel and no padding, stacking two layers gives a receptive field of 5-by-5, and stacking three layers gives 7-by-7. Using dilated convolutions, which space out the filter positions rather than sampling contiguously, allows rapid receptive field growth without adding parameters.
In practice, modern networks balance kernel size, dilation, and layer depth to match the typical scale of features in their target domain. For fine-grained texture classification, shallow networks with small receptive fields may suffice, while object detection benefits from receptive fields that cover a significant fraction of the image.