Convolutional Neural Networks Explained: How Computers Actually 'See' an Image

Quick Answer

A CNN differs from a plain fully-connected network in one core mechanism: instead of connecting every input pixel to every neuron, it slides a small learned matrix (a kernel) across the image, computing a weighted sum at each position to produce a feature map. That kernel is reused at every position (parameter sharing), which is what lets the network detect a pattern like an edge no matter where it appears in the image. We ran a real Sobel kernel across a real pixel array below and show the exact numbers it produces.

What's actually different from a plain feedforward network

A regular fully-connected layer treats an image as one long flattened list of numbers and connects every single pixel to every neuron in the next layer with its own independent weight. For even a small 100x100 grayscale image, that's 10,000 input values, and a first hidden layer of just 500 neurons would need 5 million independent weights, for one layer, before you've learned anything about images specifically. Two problems fall out of this immediately: the parameter count explodes with image size, and flattening throws away spatial structure, a fully-connected layer has no built-in notion that two pixels next to each other are more related than two pixels on opposite corners.

A CNN's convolutional layer fixes both problems with one mechanism: instead of one full-size weight matrix, it learns a small weight matrix, a kernel (commonly 3x3 or 5x5), and slides that same kernel across every position in the image, reusing the identical weights at each position. This is called parameter sharing, and it does two things at once: the number of learned weights depends on the kernel size, not the image size (a 3x3 kernel is 9 weights regardless of whether the image is 28x28 or 4000x4000), and because the same kernel is applied everywhere, a pattern the kernel learns to detect, like a vertical edge, gets detected no matter where in the image it appears. That second property, called translation invariance, is exactly what a flattened fully-connected layer doesn't get for free, it would have to separately learn to recognize the same edge at every possible position.

Everything else about training, backpropagation, gradient descent, loss functions, is identical to any other neural network. The only thing that changes is the layer's internal operation: convolution instead of full matrix multiplication against every input.

The convolution operation, computed by hand on real numbers

Descriptions of convolution usually stay abstract ("slide a filter and multiply"). Here's the actual computation, done in NumPy, on a synthetic 6x6 grayscale image with a clear vertical edge down the middle, dark pixels (value 10) on the left half, bright pixels (value 200) on the right half:

image = np.array([
    [10, 10, 10, 200, 200, 200],
    [10, 10, 10, 200, 200, 200],
    [10, 10, 10, 200, 200, 200],
    [10, 10, 10, 200, 200, 200],
    [10, 10, 10, 200, 200, 200],
    [10, 10, 10, 200, 200, 200],
])

sobel_x = np.array([
    [-1, 0, 1],
    [-2, 0, 2],
    [-1, 0, 1],
])

The Sobel-X kernel above is a classic, hand-designed edge detector (a CNN would normally learn a kernel like this from data via backpropagation, but using a known one here makes it possible to verify the output by hand). At each position, you overlay the 3x3 kernel on a 3x3 window of the image, multiply each kernel weight by the pixel underneath it, and sum all nine products into a single output value, that's one dot product per position, then slide the kernel one pixel over and repeat.

We computed this for real, both with a hand-rolled loop and independently cross-checked against NumPy's own sliding_window_view, and the two matched exactly. Here's the actual 4x4 output feature map (a 6x6 image with a 3x3 kernel and no padding produces a (6-3+1) x (6-3+1) = 4x4 output, more on that formula shortly):

Feature map after Sobel-X:
[[  0. 760. 760.   0.]
 [  0. 760. 760.   0.]
 [  0. 760. 760.   0.]
 [  0. 760. 760.   0.]]

Notice what happened: the columns sitting entirely inside the dark region or entirely inside the bright region produced 0, because there's no left-to-right intensity change under the kernel there, dot product cancels out. The two middle columns, which straddle the actual edge, produced 760, a strong response. That's the entire point of a convolution kernel proven in actual numbers: the Sobel-X kernel doesn't detect brightness, it detects a specific pattern, a left-to-right intensity change, and it lights up exactly where that pattern occurs and stays silent everywhere else, regardless of whether the surrounding region was dark or bright.

You can verify the 760 by hand too. The window covering image columns 1 to 3 (values 10, 10, 200 in each of the three kernel rows) against Sobel-X gives, per row, (-1)(10) + (0)(10) + (1)(200) = 190, and the middle row is weighted double: (-2)(10) + (0)(10) + (2)(200) = 380. Summing all three rows: 190 + 380 + 190 = 760, exactly matching the computed feature map. Nothing hidden, it's arithmetic, applied identically at every position.

Kernels are direction-sensitive, and that's the whole idea

To prove kernels detect a specific pattern and not just "any edge," we ran a second kernel, Sobel-Y, the same idea rotated 90 degrees, designed to detect horizontal edges (a top-to-bottom intensity change) instead of vertical ones, against the exact same image:

sobel_y = np.array([
    [-1, -2, -1],
    [ 0,  0,  0],
    [ 1,  2,  1],
])

Feature map after Sobel-Y:
[[0. 0. 0. 0.]
 [0. 0. 0. 0.]
 [0. 0. 0. 0.]
 [0. 0. 0. 0.]]

Every single value is exactly zero. That's the correct, expected result, and it's a good sanity check that the operation is doing what it claims: our test image has zero top-to-bottom intensity change (every row is identical, the edge only runs vertically), and Sobel-Y is specifically blind to anything except top-to-bottom change. Two different, real kernels, run against the identical input, gave completely different, specific outputs, exactly because each kernel is tuned to a specific spatial pattern rather than to "interesting-ness" in general.

In a real, trained CNN, nobody hand-designs Sobel kernels. The kernel's actual numeric weights start random and are adjusted by backpropagation just like any other weight in the network, the loss function's gradient tells each kernel weight which direction to move to reduce error on the training data. What a trained first-layer kernel ends up looking like, in practice, often resembles simple edge and color-contrast detectors like Sobel, not because anyone told it to, but because detecting edges turns out to be a genuinely useful first step for almost any image task, and gradient descent finds it on its own.

Stacking feature maps: from edges to shapes to objects

A real convolutional layer doesn't apply just one kernel, it applies many, typically 32, 64, or more in a single layer, each an independently learned 3x3 or 5x5 matrix, each producing its own feature map exactly like the ones above. Stack all of those feature maps together and you get a 3D volume: width, height, and a channel dimension equal to the number of kernels in that layer. Where the original input had 3 channels (red, green, blue), the first convolutional layer's output might have 32 channels, each one representing the presence or absence of one specific learned pattern at every spatial position.

The next convolutional layer then convolves across all of those input channels simultaneously (a kernel in a deeper layer is not 3x3, it's 3x3xC where C is however many channels came out of the previous layer), meaning it can combine multiple detected patterns, say, several different edge orientations found in the previous layer, into a slightly more complex pattern, like a corner or a curve. Stack several convolutional layers and this compounds: edges combine into textures and simple shapes, textures and shapes combine into object parts, object parts combine into whole recognizable objects, entirely through the same convolution-and-combine mechanism repeated with different learned kernels at each depth.

This is the real answer to "why does a CNN work better than a fully-connected network for images": it isn't that convolution is a fundamentally more powerful mathematical operation, it's that the hierarchy (local patterns combining into larger patterns, with weights shared across every spatial position) matches how visual structure actually works, and it does so with vastly fewer parameters than trying to let a fully-connected layer discover the same hierarchy on its own from a flattened pixel list.

Pooling: shrinking the map while keeping what matters

After a convolution produces a feature map, CNNs typically apply a pooling operation, most commonly max pooling, which slides a small window (usually 2x2) across the feature map and keeps only the maximum value in each window, discarding the rest, with no learned weights involved at all. We ran this on our own Sobel-X feature map from earlier, taking the absolute value first (since we only care about edge strength, not direction of the intensity change) and pooling with a 2x2 window and a stride of 2:

abs(feature map):
[[  0. 760. 760.   0.]
 [  0. 760. 760.   0.]
 [  0. 760. 760.   0.]
 [  0. 760. 760.   0.]]

after 2x2 max pooling:
[[760. 760.]
 [760. 760.]]

The 4x4 map shrank to a 2x2 map, one quarter the size, while the important signal, the fact that there's a strong edge response nearby, survived intact in every pooled output. This gives two practical benefits at once: it cuts the amount of data flowing into the next layer roughly by a factor of four (for a 2x2 pool), directly reducing compute and memory for everything downstream, and it gives a small amount of translation tolerance, if the edge in the original image had been one pixel to the left or right, several of the pooled output values would come out identical, because the max within each window absorbs small positional shifts.

The trade-off is a loss of precise spatial information, pooling can't tell you exactly which of the 4 pixels in a window had the maximum value once you're looking at the pooled output alone. That's an acceptable trade for most classification tasks ("is there a cat somewhere in this image" doesn't need pixel-precise edge locations), which is why pooling layers are standard in CNN designs aimed at classification, and often left out or replaced with strided convolutions in architectures where precise spatial location matters more, like segmentation.

Output size, stride, and padding: the formula behind the numbers above

The output feature map from a convolution is always smaller than the input, unless you explicitly compensate for it, because a kernel needs room to sit fully inside the image at every position. For a square input of size W, a square kernel of size K, stride S (how many pixels the kernel moves between positions), and P pixels of zero-padding added around the border, the output size is:

output_size = ((W - K + 2*P) / S) + 1

Our own example used W=6, K=3, S=1 (the default, move one pixel at a time), P=0 (no padding, called "valid" convolution): (6 - 3 + 0)/1 + 1 = 4, which is exactly the 4x4 output we actually computed above, confirming the formula against a real, hand-verified result rather than just quoting it.

Increasing the stride to 2 would skip every other position, roughly halving the output dimensions and the compute cost, at the price of a coarser feature map, useful in deeper layers where you want to downsample without adding a separate pooling layer. Adding padding (typically zeros) around the border lets you counteract the shrinkage entirely, choosing P so the output size equals the input size is called "same" padding, which matters when you want to stack many convolutional layers without the feature map shrinking to nothing before it reaches the network's later stages, and also so that pixels near the border get looked at by roughly as many kernel positions as pixels in the center, which a purely valid convolution shortchanges.

Frequently Asked Questions