What you'll learn
Quick Answer
A load balancer sits in front of several servers and distributes incoming requests between them. It also removes unhealthy servers from rotation, which makes it as much an availability tool as a scaling one.
What it does
One server handles a certain load. Beyond that you have two options: a bigger server (scaling up), which has a ceiling and a single point of failure, or more servers (scaling out), which needs something to distribute the work.
That something is the load balancer. Clients connect to one address; it forwards each request to one of several identical servers behind it.
The availability benefit is at least as important as the capacity one. If a server crashes, the load balancer notices and stops sending it traffic — users see nothing. Without one, a crash is an outage.
It also gives you a place to do deployments without downtime: remove a server from rotation, update it, add it back, repeat.
How it decides where to send a request
- Round robin — each server in turn. Simple, and fine when servers are identical and requests cost roughly the same.
- Least connections — whichever server currently has fewest active connections. Better when request duration varies a lot, since round robin can pile long requests onto one server.
- Weighted — a more powerful server receives proportionally more. Useful with mixed hardware.
- IP hash — the client's IP determines the server, so a given client consistently reaches the same one. This is one answer to the session problem below, and it distributes unevenly.
Round robin is the sensible default. Reaching for something cleverer without evidence that request cost varies is premature.
Health checks, and getting them right
The load balancer periodically requests a health endpoint from each server. Failures beyond a threshold remove it from rotation; recovery puts it back.
The design of that endpoint matters more than people expect.
Too shallow — returning 200 unconditionally means a server whose database connection is dead still receives traffic, and every request fails. The check passes while the server is useless.
Too deep — checking every dependency means a slow third-party API marks all your servers unhealthy simultaneously, and the load balancer removes every server. A partial outage becomes a total one.
The usual compromise is to check what the server needs to do its own job — its database connection — and not what its dependencies need. Some systems separate a liveness check (is the process alive) from a readiness check (can it serve traffic).
The sticky session problem
This is where scaling out breaks a working application, and it catches almost everyone once.
A user logs in. The server stores the session in memory. Their next request goes to a different server, which has never heard of them, so they appear logged out. The symptom is maddening: users randomly logged out, working fine on your single-server development machine.
Three solutions, in increasing order of quality:
- Sticky sessions — the load balancer sends a given user to the same server every time. It works and it undermines the point: that server's failure loses those sessions, and load distributes unevenly.
- Shared session store — sessions in Redis rather than memory, so any server can serve any user. The common answer.
- Stateless authentication — a signed token the client sends each time, so no server-side session exists. See API authentication explained.
The general principle: servers behind a load balancer should be stateless. Anything stored in memory or on local disk — sessions, uploaded files, caches — must move to shared storage, or behaviour depends on which server you happened to reach.
In system design interviews
A load balancer appears in almost every system design answer, usually drawn without comment. The follow-ups are where the discussion happens.
"What if the load balancer fails?" It becomes the single point of failure, so it is run redundantly, typically with a floating address that fails over to a standby. Managed cloud load balancers handle this for you.
"How do you handle sessions?" The question above. Mentioning it unprompted signals you have actually deployed something.
"Layer 4 or layer 7?" Layer 4 balances on IP and port without inspecting content — fast and simple. Layer 7 reads the HTTP request and can route by path, so /api goes to one pool and /images to another. Layer 7 is more capable and slightly more expensive.
Worth adding: a load balancer is frequently also where TLS is terminated and where rate limiting sits, since it is the one place every request passes through.