What you'll learn
Quick Answer
A neural network is a stack of simple weighted-sum-plus-nonlinearity calculations, called neurons, arranged in layers so the output of one layer feeds into the next. Every deep learning model you've heard of — the model behind a chatbot, an image recognizer, a recommendation engine — is this same idea repeated at massive scale with different layer arrangements. Learning is just gradient descent nudging millions of weight numbers, layer by layer, so the network's output error gets smaller, computed efficiently through an algorithm called backpropagation.
A Single Neuron Is Just Arithmetic
A neuron takes a list of numbers in, multiplies each one by a weight, adds them up along with a bias term, and passes the result through a small nonlinear function called an activation function. That's the entire unit. Here's one actual neuron computed by hand with numpy, with a 3-number input x = [0.5, -0.2, 0.8] feeding a tiny 2-neuron hidden layer and then a single output neuron:
def sigmoid(z):
return 1 / (1 + np.exp(-z))
x = np.array([0.5, -0.2, 0.8])
W1 = np.array([[0.1, 0.4, -0.3],
[0.2, -0.5, 0.6]])
b1 = np.array([0.1, -0.1])
W2 = np.array([0.7, -0.9])
b2 = 0.05
z1 = W1 @ x + b1
a1 = sigmoid(z1)
z2 = W2 @ a1 + b2
y_hat = sigmoid(z2)Running this gives z1 = [-0.17, 0.58] (the raw weighted sums), a1 = [0.4576, 0.6411] (after squashing through sigmoid), z2 = -0.2066, and a final output y_hat = 0.4485. That's the whole forward pass: two matrix multiplications and two applications of a squashing function. Everything else in a neural network — however many layers, however many millions of numbers — is this same operation done more times, at bigger scale.
Stacking Neurons Into Layers
One neuron can only draw a straight-line decision boundary — it's the same shape as basic linear regression. The reason networks are useful is that stacking layers of neurons lets each layer build on features the previous layer detected, so the network as a whole can represent curved, complicated decision boundaries that no single neuron could. In the example above, the hidden layer has 2 units; a real network solving a real problem might have hundreds of units per layer and dozens of layers, but the mechanism at each individual neuron is identical to the one you just saw computed by hand.
The number of learnable numbers (weights and biases) grows fast: the tiny network above has (2×3 + 2) + (2×1 + 1) = 11 parameters. A modest image classifier can have millions. A large language model has billions. None of that changes what a parameter is — it's still just a number multiplied against an input, adjusted by training so that the network's output gets closer to the correct answer.
What depth buys you specifically is composability: an early layer in an image model might learn to detect edges, the next layer combines edges into shapes, and a later layer combines shapes into object parts — each layer's output becoming the next layer's raw material, the same way a1 became the input to the output neuron above.
Why Nonlinear Activation Functions Matter
The activation function isn't decorative — without it, depth would be pointless. A matrix multiplication followed by another matrix multiplication is still just one big matrix multiplication; stacking purely linear layers collapses mathematically into a single linear layer, no matter how many you stack. The nonlinearity in between is what actually lets each additional layer represent something the previous layer couldn't.
Three activation functions show up constantly, each with a real tradeoff:
- Sigmoid squashes any input into the range 0 to 1, which made it a natural first choice and is still useful for a final output that needs to look like a probability. Its problem is that for large positive or negative inputs, the curve goes nearly flat — the gradient there is close to zero, which means a network with many sigmoid layers struggles to propagate a learning signal back through all of them. This is the classic vanishing gradient problem.
- Tanh is sigmoid's cousin, squashing into -1 to 1 instead of 0 to 1, which centers outputs around zero and often trains a bit better in practice, but has the same flattening problem at the extremes.
- ReLU (Rectified Linear Unit) is simply
max(0, z)— it's nearly free to compute and doesn't flatten out for positive inputs, which is why it became the default for hidden layers in most modern networks. Its own failure mode is the "dying ReLU" problem: if a neuron's weights end up always producing a negative input, its gradient is permanently zero and it stops learning entirely.
How a Network Learns: Loss and Backpropagation
Training means adjusting every weight so the network's output gets closer to the correct answer, measured by a loss function. Using the tiny network above with a target output of 1 and squared-error loss, loss = 0.5 × (y_hat - target)², the current loss is 0.5 × (0.4485 - 1)² = 0.1521. Gradient descent's job is to figure out, for every single weight, whether nudging it up or down makes that loss number smaller, and by how much — that sensitivity is the gradient.
Backpropagation is the algorithm that computes all of those gradients efficiently, using the chain rule from calculus to work backward from the loss through each layer. Rather than re-deriving that by hand, it's worth actually checking that it gives the right answer: computing the gradient analytically via backprop for every weight in the tiny network above, then comparing it against a numerical estimate (nudging each weight by a tiny amount and measuring how much the loss actually moved):
delta2 = (y_hat - target) * y_hat * (1 - y_hat)
dL_dW2 = delta2 * a1
delta1 = delta2 * W2 * a1 * (1 - a1)
dL_dW1 = np.outer(delta1, x)The analytic result was dL/dW2 = [-0.062421, -0.087447] and dL/dW1 = [[-0.01185, 0.00474, -0.01896], [0.014124, -0.00565, 0.022599]]. Comparing those against gradients estimated by nudging each weight by 0.000001 and measuring the loss change directly gave the same numbers to 5 decimal places, with a maximum difference of about 0.00000000003 across every weight. That match is exactly why backpropagation works: it's computing the true mathematical gradient, just far more efficiently than checking each weight one at a time — for a network with a million weights, backprop gets you all million gradients in roughly the cost of two passes through the network, not a million separate nudge-and-measure experiments.
Once you have the gradient for a weight, gradient descent updates it by moving a small step in the opposite direction: weight = weight - learning_rate × gradient. Do that for every weight, for every batch of training examples, repeatedly, and the loss trends downward — that repeated process, done over the whole dataset many times, is what "training" means.
Watching a Real Network Learn
To see this actually happen rather than just describe it, here's a small neural network — scikit-learn's MLPClassifier with one hidden layer of 8 units — trained on a synthetic two-moons dataset (1,200 points, 30% held out for testing), tracked every few training epochs:
clf = MLPClassifier(hidden_layer_sizes=(8,), activation='tanh',
solver='sgd', learning_rate_init=0.5)The real recorded accuracies were:
epoch 1 5 10 20 40 80 150 300
train 0.820 0.876 0.879 0.905 0.929 0.932 0.931 0.933
test 0.797 0.850 0.847 0.878 0.908 0.911 0.914 0.908Both curves climb together and level off around 92-93%, with the gap between them staying small the entire time — training accuracy never runs away from test accuracy. That's exactly what you want to see, and it's not automatic: an 8-unit hidden layer happens to be a reasonable amount of capacity for a problem this simple. Our companion post, Overfitting Explained, uses this exact idea of tracking train versus validation performance to show what it looks like when capacity and data size are mismatched, and the two curves pull apart instead of moving together.
The final training loss (scikit-learn's log-loss) after 300 epochs was 0.166 — a real, non-zero number, because with tanh activations, SGD, and this much noise deliberately mixed into the data, the model was never going to reach zero loss, and it shouldn't.
Common Architectures, in One Line Each
Everything above describes a plain feedforward network — every neuron in one layer connects to every neuron in the next. Most named architectures are really just structured variations on that same neuron-plus-activation idea, constrained in a way that fits a specific kind of data:
- CNNs (Convolutional Neural Networks) reuse the same small set of weights, sliding them across an image, so the network learns a pattern detector once and applies it everywhere rather than learning a separate detector for every pixel position — this is what makes them practical for images.
- RNNs and LSTMs process a sequence one step at a time, carrying a running summary forward, which made them the default for text and time series before a better mechanism came along.
- Transformers replaced that step-by-step processing with an attention mechanism that lets every position in a sequence look directly at every other position at once. This is the architecture behind modern language models, and it's covered in depth in our How LLM APIs Work post rather than repeated here.
The point worth remembering is that none of these invent a new kind of neuron — they're different rules for how neurons connect to each other, layered on top of the exact weighted-sum-and-activation mechanism from the first section.
Getting Started, Practically
If you're learning this for the first time, resist the pull toward building a large model before you've built a small one. A single hidden layer on a small, well-understood dataset using scikit-learn's MLPClassifier or a minimal PyTorch model will teach you more about what's actually happening than jumping straight to a pretrained deep network you can't inspect end to end.
A few habits that matter more than architecture choice early on: always hold out validation data and watch both accuracy curves, not just the training one — a single number after training tells you almost nothing on its own. Normalize your input features before training; a raw feature ranging from 0 to 10,000 next to one ranging from 0 to 1 will dominate the early gradients for reasons that have nothing to do with which feature actually matters. And get a working baseline first, then add capacity — it's much easier to diagnose a small network that's underperforming than a large one that's doing something you don't understand.