The Thundering Herd Problem: Why a Thousand Clients Can Take Down One Server at Once

Quick Answer

The thundering herd problem happens when a cache entry expires and every concurrent request notices the miss at the same instant, so all of them call the slow origin at once instead of just one of them. A simulation with 500 concurrent requests shows this literally: 500 origin calls with no coordination, versus 1 origin call once a single-flight lock is added. The fix is always the same shape: let exactly one request recompute the value while everyone else waits for it or reads a slightly stale copy.

What the Thundering Herd Problem Actually Is

The thundering herd problem is what happens when a huge number of clients all take the exact same action, at the exact same moment, against the exact same resource, and that resource cannot handle all of them arriving at once. The name is old — it originally described operating systems where several processes were all blocked waiting on one event (like a new connection arriving on a shared socket), and when that event fired, every single one of them woke up at once to compete for it, even though only one could actually win. Most would wake up, find nothing to do, and go back to sleep — but waking a thousand processes to let 999 of them discover they wasted a context switch is itself expensive.

On the web, the far more common version of this is a cache expiring. You've built a system the right way: an expensive piece of data (a database aggregate, a rendered page, a third-party API response) gets computed once and cached with a time-to-live, say 60 seconds, so the next 10,000 requests in that window hit the cache instead of recomputing it. This works beautifully right up until second 61, when the cache entry expires and every one of those concurrent requests checks the cache at nearly the same instant, every one of them sees a miss, and every one of them independently decides it is responsible for recomputing the value and calling the origin. A system that was serving 10,000 requests a second off a warm cache suddenly fires 10,000 origin calls in the same fraction of a second. The origin, which was never sized to handle that, falls over.

This is distinct from a traffic spike or a DDoS attack, where the volume of incoming requests itself is the problem. In a thundering herd, the incoming request rate hasn't changed at all — what changed is that a protective layer (the cache) that was absorbing almost all of that traffic suddenly let all of it through at once, for one brief window, because it had no way to coordinate the many clients who were all about to do the same expensive thing.

The Classic Trigger: A Cache Entry Expiring Under Load

The mechanics are worth being precise about, because the fix depends entirely on understanding exactly where the coordination is missing. A typical read path looks like: check the cache, if present and fresh return it, if absent or expired go compute the real value from the origin (a database query, an upstream API call, a heavy computation), store it back in the cache, return it. Each individual request, read in isolation, is doing something completely reasonable. The problem only exists in the gap between many requests arriving concurrently and each one running that same logic independently, with no awareness that hundreds of other requests are doing the identical thing in parallel.

This gets worse, not better, under load, because cache misses are precisely correlated with your busiest moments. A low-traffic endpoint that gets one request every few seconds will almost never have two concurrent misses — by the time the second request arrives, the first one has already finished repopulating the cache. A hot endpoint doing thousands of requests per second has the opposite problem: by the time any single request's cache-miss recompute finishes, hundreds more requests have already arrived, seen the same miss, and started their own redundant recompute. The busier the system, the more damage one expiry event causes, which is exactly the opposite of what you want from a caching layer.

Proving It: A Real Simulation With No Coordination

Rather than just asserting this happens, it's worth actually simulating it, because the numbers are the whole point. Below is a real cache class with no coordination between requests, plugged into a slow "origin" function that stands in for a database query or upstream call. 500 threads are lined up on a barrier so they all call .get() at essentially the same instant, right after the cache entry has just expired:

class NaiveCache:
    def __init__(self, ttl):
        self.ttl = ttl
        self.value = None
        self.expires_at = 0

    def get(self):
        now = time.time()
        if self.value is not None and now < self.expires_at:
            return self.value, "HIT"
        # cache miss (or expired) -> go to origin, no coordination at all
        result = slow_origin_compute()
        self.value = result
        self.expires_at = now + self.ttl
        return result, "MISS"

Running this with 500 concurrent threads against a cache that just expired gives a real, measured result: all 500 of them register a cache MISS, and the origin function is called 500 times (501 counting the one priming call used to populate the cache before the test). Each of those calls does the expensive work from scratch, at the same moment, against whatever the origin actually is. If the origin is a database that can comfortably handle 20 concurrent heavy queries, 500 of them arriving together is how you get a connection pool exhaustion error and a cascading outage, not just a slow response.

The Fix: A Single-Flight Lock

The standard fix is called single-flight (or request coalescing, or sometimes a "dogpile lock"): when a request sees a miss, before it calls the origin, it must acquire a lock scoped to that specific cache key. Only one request gets the lock and actually calls the origin. Every other request that arrives during that window blocks on the same lock instead of independently hitting the origin, and once the first request finishes and releases the lock, everyone who was waiting re-checks the cache (now freshly populated) and gets a normal hit. This is the exact same re-check-after-acquiring-the-lock pattern as double-checked locking in any other concurrency context, applied specifically to cache repopulation:

class SingleFlightCache:
    def __init__(self, ttl):
        self.ttl = ttl
        self.value = None
        self.expires_at = 0
        self.lock = threading.Lock()

    def get(self):
        now = time.time()
        if self.value is not None and now < self.expires_at:
            return self.value, "HIT"

        # Only one thread gets to hold the lock and recompute.
        with self.lock:
            # Re-check after acquiring the lock: someone else may have
            # already refreshed the value while we were waiting.
            now = time.time()
            if self.value is not None and now < self.expires_at:
                return self.value, "HIT-AFTER-WAIT"
            result = slow_origin_compute()
            self.value = result
            self.expires_at = now + self.ttl
            return result, "MISS-RECOMPUTED"

Running the identical 500-thread experiment against this version gives a dramatically different, equally real result: 499 of the 500 threads get HIT-AFTER-WAIT, and exactly 1 gets MISS-RECOMPUTED. Total origin calls drop from 501 to 2 (one priming call, one real recompute during the test) — a reduction of 499 origin calls for the same 500 concurrent requests, with every single request still getting a correct, fresh value back. That's not a theoretical improvement; it's the measured output of running both versions against the same simulated load.

Other Mitigations You'll See in Production

A single global lock per key is the simplest fix, but production systems reach for a few variations depending on what they can tolerate:

• Stale-while-revalidate: instead of making the waiting requests block, serve them the slightly-expired old value immediately while exactly one background request refreshes it. Readers never wait at all; they just occasionally get a value that's a few hundred milliseconds stale, which is usually a fine trade.
• Probabilistic early expiration (sometimes called XFetch): instead of every request treating the TTL as a hard deadline, each request computes a small random chance of refreshing early, with that chance increasing the closer the entry is to actually expiring. This spreads the recompute across many individual requests over time instead of letting it all collide on one instant.
• Jittered TTLs: if you cache many different keys that were all populated at the same deploy or cache-warm event, give each one a TTL with a bit of randomness added (e.g. 60 seconds plus or minus 10) so they don't all expire in the same second and create a herd across many keys at once, not just one.
• Grace mode / serve-from-stale on origin failure: if the origin call does fail under load, keep serving the last known-good cached value rather than returning an error to every client, buying time for the origin to recover.

Caching layers and tools often bake one of these in: Varnish's grace setting, memcached client libraries that support a "gets" token for optimistic locking on repopulation, and Go's golang.org/x/sync/singleflight package are all production implementations of the same core idea demonstrated above.

How This Differs From Distributed Locks and Rate Limiting

It's worth being precise about how this relates to two other topics that sound similar. A distributed lock is the general-purpose primitive — a way to let exactly one process in a fleet do something at a time, coordinated across machines, usually via Redis, ZooKeeper, or similar. The single-flight lock in this post is one specific application of that same idea, scoped tightly to "only one request may repopulate this one cache key right now." Rate limiting solves a different problem: it caps how many requests a client or endpoint is allowed to make in a window, which helps if the herd is self-inflicted by one misbehaving client hammering you, but does nothing by itself if the herd is many different legitimate clients all reacting to the same cache expiry at once — rate limiting the origin doesn't stop 500 requests from trying, it just decides which of the 500 get rejected. Single-flight coalescing is specifically about making sure only one of those legitimate requests ever reaches the origin in the first place.

When You'll Actually Run Into This

You don't need 500 real concurrent users to hit this in production; a handful of common, everyday events can trigger it:

• A cache or CDN edge node restarts or redeploys, losing its warm cache entirely, while traffic to it continues at the normal rate.
• A popular page's cache entry expires during its own traffic peak (a product page during a flash sale, a news article right as it goes viral) — the worst possible timing, and also the most likely, because popular things get hit constantly.
• A backend service restarts and every client in a connection pool reconnects at the same instant, which is the same herd shape applied to connections instead of cache misses.
• A scheduled job (a cron-triggered cache flush, a deploy that clears a cache namespace) deliberately invalidates many keys at once, creating a herd across all of them simultaneously rather than one key at a time.

In every one of these, the fix is the same shape: stop treating "the cache is empty" as something every request handles independently, and make exactly one code path responsible for refilling it while everyone else either waits briefly or reads the old value a little longer.

Frequently Asked Questions