What you'll learn
Quick Answer
A flaky test passes and fails on identical code across repeated runs, usually because it depends on something outside the test's control, such as real time, unseeded randomness, or a race with an async operation. The fix is to remove that dependency using fake timers, seeded or injected randomness, and properly awaited async code, not retrying until it happens to go green.
What "flaky" actually means
A flaky test is one that passes and fails on the same code, without anyone changing anything, across repeated runs. That is the whole definition. It is not about the test being wrong in the way a bug is wrong, it is about the test being non-deterministic: its outcome depends on something other than the code it claims to verify, like the exact wall-clock timing of the run, the specific value a random number generator happened to produce, or the order other tests ran in.
Flaky tests are expensive in a specific way: they erode trust. Once a team learns that a red build sometimes means nothing, people start re-running failed pipelines instead of reading them, and eventually a red run gets ignored right up until it represents a real bug that ships. "Run it again until it's green" treats the symptom and leaves the actual non-determinism in place for the next person.
A genuinely flaky test, run fifteen times
Here is a real flaky test, not a hypothetical one. It fires an async callback after a random delay and asserts the callback ran fast:
it('callback fires within 15ms', () => {
const start = Date.now();
return new Promise((resolve, reject) => {
processWithRandomDelay(() => {
try {
const elapsed = Date.now() - start;
expect(elapsed).toBeLessThan(15);
resolve();
} catch (err) { reject(err); }
});
});
});The delay itself is Math.random() * 30, uniform between 0 and 30ms, and the assertion demands under 15ms. Running this exact file, unmodified, fifteen times in a row produced 2 passes and 13 fails, in this order: fail, fail, fail, fail, pass, fail, fail, fail, fail, pass, fail, fail, fail, fail, fail. One failing run reported AssertionError: expected 36 to be less than 15, a real number from a real setTimeout. Nothing about the source code changed between any of these runs. That is what flaky actually looks like: not an occasional CI hiccup, but a coin flip built directly into the test.
The usual suspects behind flaky tests
The delay example combines two of the most common root causes, and it is worth naming all of them:
- Unseeded randomness. Math.random(), or any RNG without a fixed seed, produces a different value every run, and any assertion downstream inherits that unpredictability.
- Real time and timers. Asserting against wall-clock duration or a tight setTimeout window makes the test sensitive to machine load, and a busy CI runner behaves differently from a laptop.
- Unawaited async work. A test that does not properly wait for a promise or callback can pass or fail depending on which finishes first, effectively racing the code under test.
- Shared state between tests. A previous test leaving data in a database, a global variable, or a file makes a later test's outcome depend on run order.
- External dependencies. A real network call or a real third-party API introduces latency, rate limits, and outages that have nothing to do with the code being tested.
All five share one trait: something in the test depends on a source of variation the test itself does not control.
The fix: remove every source of real-world variation
The fix is to replace every real-world dependency with a fixed, controlled one. For the delay test: fake the clock, and inject a fixed value in place of the random one.
it('callback fires with the expected delay (deterministic)', async () => {
const fixedRandom = () => 0.1; // stands in for a seeded RNG
const promise = new Promise((resolve) => {
processWithRandomDelay((delay) => resolve(delay), fixedRandom);
});
await vi.advanceTimersByTimeAsync(3);
const delay = await promise;
expect(delay).toBe(3);
});vi.useFakeTimers() replaces the real setTimeout with one the test controls directly, and the injected fixedRandom function replaces Math.random() with a known value. The assertion now checks an exact, predictable number instead of racing a real clock. Run this version fifteen times in a row, same as before: 15 passes, 0 fails, every single time. Nothing about the underlying behaviour changed, the code still schedules a callback and resolves a delay, only the test stopped depending on the real clock and the real random generator to get there.
Retries hide problems, they don't fix them
Reaching for a retry plugin instead of a fix is the most common shortcut, and it is usually the wrong one. Auto-retrying a failing test until it goes green hides two different problems equally well: a test that is genuinely non-deterministic, and a real race condition in the application that will eventually surface in production the same way it did in the test. Before configuring retries, spend ten minutes asking whether the failure pattern looks like the app or the test. A test racing its own timers is the pattern above; a service call intermittently returning the wrong data is a more serious problem wearing a flaky test's clothes.
If a flaky test cannot be fixed immediately, quarantine it explicitly, skip it with a comment and a ticket number, not silently, so its failures stop polluting the build without pretending the gap does not exist. And if a test only ever verified something trivial, deleting it is a legitimate outcome; a flaky test that protects nothing important is not worth the ongoing cost of fixing.