Idempotency: Making It Safe to Retry

Quick Answer

An operation is idempotent if performing it several times has the same effect as once. It matters because network failures leave clients unable to tell whether a request succeeded, so retries are unavoidable.

The problem: an ambiguous failure

A client sends a request to charge a card. The connection times out with no response.

What happened? Three possibilities, indistinguishable from the client's position:

• The request never arrived. Nothing happened.
• It arrived, the charge succeeded, and the response was lost on the way back.
• It arrived and failed partway through.

The client cannot tell. If it retries, it may charge twice. If it does not, the customer may not be charged at all and the order is inconsistent.

This is not an edge case — it is the normal behaviour of networks. Timeouts, dropped connections and lost responses happen continuously at scale.

Idempotency removes the dilemma. If retrying is guaranteed harmless, the client simply retries, and the ambiguity stops mattering.

What is naturally idempotent

Some operations already are, and recognising which is half the work.

• Idempotent: GET a resource. DELETE a specific record — deleting an already-deleted thing leaves the same end state. Setting a value: status = 'shipped' repeated changes nothing after the first.
• Not idempotent: POST creating a new record — each call creates another. Relative changes: balance = balance - 100 subtracts every time. Appending to a list. Sending an email.

The pattern: absolute assignment is idempotent, relative modification is not. "Set the quantity to 5" is safe to repeat; "add 5 to the quantity" is not.

Where you can express an operation absolutely, do — it removes the problem entirely rather than requiring machinery to manage it.

Idempotency keys

For genuinely creative operations, the standard solution is a client-generated key.

The client generates a unique value per logical operation and sends it with the request:

POST /payments
Idempotency-Key: 9f2c1ab4-...

{ "order_id": "ord_99", "amount": 19900 }

The server records the key with the result. On seeing a key it has already processed, it returns the stored response rather than performing the operation again.

So a retry with the same key is safe: the client gets the original result, and the charge happened once. This is how payment providers handle exactly this problem, and it is worth knowing by name.

Two implementation details that matter. The key must be generated per operation and reused across retries of that operation — a fresh key on each retry defeats the entire mechanism. And the record must be written in the same transaction as the effect, or two concurrent retries can both pass the check before either has recorded anything.

The concurrency trap

The naive implementation looks correct and is not:

if already_processed(key):      # check
    return stored_response
result = charge_card()          # act
store(key, result)              # record

Two retries arriving simultaneously can both pass the check before either records. Both charge. The window is small and, under load, it is hit.

The fix is to make the database enforce it. Insert the key with a unique constraint first, and let the second insert fail:

try:
    insert_key(key)             # unique constraint
except UniqueViolation:
    return wait_for_stored_response(key)
result = charge_card()
store_response(key, result)

Now the database — which can enforce atomicity — decides, rather than application code with a gap between checking and acting. This is the same reasoning as preventing double bookings, and the same class of bug as a race condition in threading.

Where this comes up

• Payment APIs — the canonical case, and where idempotency keys were popularised.
• Message queue consumers. Most queues deliver at least once, so handlers must tolerate duplicates — see message queues.
• Webhook receivers. Providers retry when they do not get a prompt success, so the same event arrives twice — see webhooks.
• Scheduled jobs, which may overlap or be re-run manually.
• Form submissions. A user double-clicking submit is the same problem at human speed. Disabling the button helps the honest case; the server must still be safe.

The general principle worth carrying: in any distributed system, assume every message may arrive more than once, and design so that it does not matter. Attempting to guarantee exactly-once delivery is far harder than making the handler idempotent, which is why the industry standardised on the latter.

Frequently Asked Questions