Chaos Engineering Explained: Breaking Things on Purpose to Build Confidence

Quick Answer

Chaos engineering is the practice of deliberately injecting failure into a system — killing a server, adding network latency, exhausting a resource — in a controlled way, to find weaknesses before they cause a real outage. It started at Netflix with a tool called Chaos Monkey, which randomly terminates production instances, and it's grown into a discipline with a formal methodology: form a hypothesis about steady-state behavior, inject a real-world failure, and compare what actually happens against what you expected.

What Chaos Engineering Actually Is

Most testing verifies that a system does what you expect under conditions you designed the test around. Chaos engineering does the opposite: it deliberately introduces conditions you didn't design around — a server disappearing mid-request, a network call taking 10 seconds instead of 50 milliseconds, a disk filling up — specifically to find out whether the system's actual behavior matches the resilience you assumed it had. The premise is uncomfortable but accurate: distributed systems fail in ways nobody fully predicted, and the only reliable way to find those ways is to cause failures under controlled conditions, on your own schedule, with your best engineers watching, rather than waiting for the failure to happen unannounced at 2 a.m. during a real incident.

This is different from traditional fault-tolerance testing in an important way: chaos engineering experiments are typically run against production, or a system as close to production as possible, because that's the only environment where the real interactions between real traffic patterns, real configuration drift, and real infrastructure quirks actually exist. A staging environment that's smaller, more lightly loaded, and configured slightly differently from production routinely fails to reproduce the exact conditions under which a resilience mechanism (a retry policy, a circuit breaker, a fallback) actually gets exercised.

The discipline has a name specifically because it isn't ad hoc "let's see what breaks" button-mashing. It follows an actual experimental method, borrowed loosely from the scientific method: define what normal, healthy behavior looks like, form a hypothesis about how the system will respond to a specific injected failure, run the experiment, and compare the real outcome to the hypothesis.

Where This Came From: Netflix and Chaos Monkey

Chaos engineering as a named discipline traces directly back to Netflix's migration to AWS in the late 2000s and early 2010s. Running on cloud infrastructure meant individual virtual machine instances could, and regularly did, disappear without warning — a fundamentally different failure mode than the physical servers Netflix had run in its own data centers before. Rather than hoping their services were resilient to random instance loss, Netflix built Chaos Monkey, a tool whose entire job is to randomly terminate instances in production, during business hours, on a schedule.

Chaos Monkey is open source (available on GitHub under Netflix/chaosmonkey) and it's built specifically to integrate with Spinnaker, Netflix's continuous delivery platform — Spinnaker is what actually knows how a given service is deployed and carries out the termination Chaos Monkey decides on. Teams opt individual applications into Chaos Monkey through configuration in Spinnaker itself, and the default configuration targets a mean time between terminations of about two days for an opted-in service, meaning any given instance of that service can expect to be killed roughly every couple of days on average.

The reasoning behind running this constantly, rather than as an occasional special event, is deliberate: if instance termination is a rare, dramatic event, engineers build systems that merely survive it when it happens. If instance termination is a routine, near-constant background fact of running on the platform, engineers are forced to build systems that handle it gracefully as a matter of course — because if they don't, Chaos Monkey will expose that gap in days, not months.

The Actual Methodology: Steady State, Hypothesis, Injection, Comparison

A real chaos engineering experiment has a specific shape, and skipping steps is how "chaos engineering" degrades into just breaking things randomly with no learning attached. The shape looks like this:

• Define steady state — pick a measurable signal that represents the system behaving normally, like checkout success rate, p99 API latency, or requests served per second. This has to be something you're actively measuring already, not something you'll eyeball after the fact.
• Form a hypothesis — state, before running anything, what you expect to happen to that steady-state metric when you inject a specific failure. For example: "if we kill one of three instances behind this load balancer, checkout success rate will stay above 99.9% because the other two instances absorb the load."
• Inject a real-world failure — actually cause the condition: terminate an instance, add artificial network latency between two services, throttle CPU, fill a disk, or block access to a dependency. The failure should resemble something that has happened, or plausibly could happen, in production — not an arbitrary or contrived condition.
• Compare hypothesis to reality — measure the steady-state metric during and after the injected failure and compare it against what you predicted. If it matches, you've gained real evidence the system is as resilient as you believed. If it doesn't, you've found a weakness under controlled conditions, with engineers watching and ready to intervene, instead of finding it during an unplanned outage.
• Minimize blast radius, then expand — start with the smallest experiment that could produce a useful answer (one instance, a small percentage of traffic, a low-traffic time window), and only widen scope once smaller experiments consistently confirm the hypothesis.

That last point is what separates disciplined chaos engineering from recklessness: you're not trying to see how bad you can make things, you're trying to find the smallest, safest experiment that still tells you something true about the system's real-world resilience.

Game Days: The Human Side of Chaos Engineering

Not every chaos experiment is an automated background tool quietly killing instances. A game day is a scheduled, deliberate exercise where a team gets together — often across engineering, on-call, and sometimes customer support — to run a chaos scenario live and practice the human response to it, not just observe the system's technical response. A typical game day picks a specific, plausible failure scenario ("our primary database region becomes unreachable"), simulates it in a controlled way, and then has the on-call engineers actually go through detecting it, diagnosing it, and following the runbook to mitigate it, in real time, with the rest of the team watching and taking notes.

The value of a game day is as much about testing the humans and the process as it is about testing the software. Runbooks that look complete on paper often turn out to reference a dashboard that was renamed eight months ago, or assume access to a system that the current on-call engineer doesn't actually have permissions for, or describe a mitigation step that takes twenty minutes when the team assumed it took two. None of those gaps show up by reading the runbook — they only show up when someone actually tries to follow it under simulated pressure, which is exactly what a game day forces.

Game days are also where chaos engineering becomes organizationally sustainable. A background tool silently killing instances can quietly get disabled or ignored if nobody's paying attention to its findings. A game day is a scheduled event with visible participation and a debrief afterward, which builds the habit and the institutional muscle memory needed to keep taking resilience seriously over time, not just during the initial adoption enthusiasm.

What Actually Gets Injected: Common Failure Types

• Instance or process termination — the original Chaos Monkey technique: kill a running instance or container and observe whether traffic reroutes cleanly to healthy instances.
• Network latency and packet loss — artificially slow down or drop a percentage of network calls between two specific services, to test timeout and retry behavior under degraded (not fully down) conditions, which is a more common real-world failure mode than a clean, total outage.
• Dependency failure simulation — make calls to a specific downstream service or third-party API fail or time out, to test whether fallbacks, circuit breakers, and bulkheads actually engage the way they're designed to.
• Resource exhaustion — deliberately consume CPU, memory, or disk on a target host, to test whether the system degrades gracefully (shedding load, returning cached results) or falls over entirely.
• Region or availability-zone failure — simulate an entire cloud region or availability zone becoming unreachable, which is the largest-blast-radius category and typically reserved for mature chaos engineering practices with strong safeguards, since a mistake here can cause a genuine outage rather than a controlled one.

Tools beyond Chaos Monkey have grown up around these categories — Gremlin offers a commercial platform covering most of them, and the CNCF hosts an open-source project called Chaos Mesh aimed specifically at Kubernetes environments — but the specific tool matters far less than following the actual experiment methodology described above.

Getting Started Without Causing a Real Outage

Teams new to chaos engineering should resist the instinct to start with the most dramatic possible experiment. Start in a non-production environment if a genuinely production-like staging environment exists, but treat that as a warm-up, not the destination — the real value comes from eventually running in production, because that's the only place where real traffic patterns and real configuration exist. When moving to production, start during low-traffic windows, with the smallest possible blast radius (a single instance, a single low-priority service, a small percentage of traffic), and with an engineer ready to abort the experiment immediately if the steady-state metric degrades further than the hypothesis predicted.

It also helps enormously to have already built the resilience mechanisms this site covers elsewhere — circuit breakers, retries with backoff, and bulkheads — before running chaos experiments against a system that has none of them. Chaos engineering is a way of validating that resilience mechanisms work as intended; it's a poor way to discover for the first time that a system has none, since that discovery could come at the cost of a real, painful outage rather than a controlled experiment. Build the safety mechanisms first, then use chaos engineering to prove they actually work under real conditions rather than just in theory.

Frequently Asked Questions