Understanding Idempotency By Building It

I’ve come across the word idempotency plenty of times while working with APIs, especially around payments, webhooks, and retries. I understood the general idea: if the same request happens more than once, we should avoid accidentally performing the same operation more than once.

That definition makes sense, but I wanted to understand what the problem actually looks like in code.

So instead of starting with Redis, databases, payment providers, or a production-ready implementation, I built the smallest example I could think of: an Express server with a fake payment endpoint.

The idea was to first create the problem, then add idempotency one piece at a time until the reason for each part of the implementation became clear.

The complete source code is available on GitHub:

View the source code on GitHub

The starting-point branch contains the simple payment API before idempotency is implemented, while the master branch contains the completed version.

Starting with a deliberately broken payment API

The example starts with a very simple endpoint:

POST /payments

It accepts an amount:

{
  "amount": 100
}

and creates a fake payment.

There is no Stripe integration and there is no database. Payments are just stored in an in-memory array.

The important part is that every time createPayment() runs, we treat that as if a real financial operation took place.

So the initial behavior is straightforward.

I send:

POST /payments

{
  "amount": 100
}

and the server creates one payment.

If I send the exact same request again, it creates another payment with another ID.

That means our fake customer has now been charged $100 twice.

At first this might seem correct. We did send two HTTP requests.

The problem is that two HTTP requests do not necessarily mean the user intended to perform two different operations.

Why would the same request happen twice?

Imagine a client sends a payment request.

The server receives it and successfully processes the payment, but something goes wrong while sending the response back to the client.

The situation might look like this:

Client
  |
  | POST /payments
  v
Server
  |
  | Payment succeeds
  |
  X Response is lost

From the client’s point of view, it doesn’t know what happened.

Did the payment fail?

Did the server receive the request?

Did the payment succeed and only the response fail?

One reasonable thing for the client to do is retry the request.

Without any protection, the server receives that retry and creates another payment.

First request  -> $100 charged
Response       -> lost
Retry          -> another $100 charged

The customer only intended to make one payment, but we performed the side effect twice.

This is the problem idempotency is trying to solve.

Introducing an idempotency key

We need some way for the client to tell the server:

This request is another attempt at the same operation.

One common approach is an idempotency key.

The client generates a unique identifier for an operation and sends it with the request:

Idempotency-Key: payment-123

Our request now looks something like:

POST /payments
Idempotency-Key: payment-123
Content-Type: application/json

{
  "amount": 100
}

If the client needs to retry that operation, it sends the same idempotency key again.

A completely different payment gets another key:

payment-123 -> $100

payment-456 -> $100

Both payments happen to have the same amount, but they represent different operations.

That helped me understand an important distinction.

Idempotency isn’t about detecting requests that happen to look similar. The key identifies the logical operation the client is trying to perform.

Remembering operations we already processed

For this example, I used a JavaScript Map as a simple in-memory idempotency store.

The basic idea is:

Idempotency key
      |
      v
Previous result

When a request arrives, we check whether we have already seen its key.

If we haven’t seen it, we create the payment and save the response.

If we have seen it, we return the previous response instead of calling createPayment() again.

The flow becomes:

Request
   |
   v
Does the key exist?
   |
   +---- Yes ----> Return previous result
   |
   No
   |
   v
Create payment
   |
   v
Store result
   |
   v
Return result

Now I can send the same request twice with the same key and only one payment is created.

The second request still reaches the server. We aren’t preventing retries.

We are making retries safe.

That distinction is probably the simplest way I found to think about idempotency:

Idempotency isn’t necessarily about preventing duplicate requests. It’s about preventing duplicate requests from repeating the side effect.

The key alone isn’t enough

At this point the implementation appears to work, but there is another problem.

Imagine the first request is:

Idempotency-Key: abc123

with:

{
  "amount": 100
}

The server processes it and remembers the result.

Later, because of a bug in the client, another request arrives using the same key:

Idempotency-Key: abc123

but the body is:

{
  "amount": 500
}

If we only check the idempotency key, the server finds abc123 and returns the previous $100 payment.

That isn’t really a retry anymore.

The client is asking us to perform something different while accidentally reusing the identifier from an earlier operation.

To detect this, I added a fingerprint for the request.

For the small example, I used:

const fingerprint = JSON.stringify(req.body);

The idempotency record can now contain both the fingerprint and the response.

Conceptually:

abc123
   |
   v
{
  fingerprint,
  response
}

When we receive the key again, we compare the new request with the original one.

If both the key and request match:

Same key + same request
-> return the existing result

If the key matches but the request is different:

Same key + different request
-> reject it

In the example, I return 409 Conflict.

The behavior now becomes:

At this point I thought the implementation was basically finished.

There was still another problem.

The concurrency problem

My first implementation effectively did this:

1. Check whether the key exists
2. Perform the payment
3. Store the result

That works when requests arrive one after another.

But real operations are usually asynchronous.

A payment might require calling another service over the network, waiting for a database, or communicating with a payment provider.

To simulate that, I added a delay to the fake payment service.

That creates a window where two requests can arrive at almost exactly the same time.

Imagine this:

Request A                 Request B

Check key
Not found

                          Check key
                          Not found

Create payment            Create payment

Store result              Store result

Both requests checked the Map before either one had stored anything.

So both requests believed they were responsible for creating the payment.

I could have an idempotency key on both requests and still end up with two charges.

The Map itself wasn’t the problem.

The problem was when I wrote to it.

There was a gap between:

Check the key

and:

Store the result

and the side effect happened inside that gap.

Claiming the operation before doing the work

Instead of waiting until the payment finishes before storing anything, the first request should claim the key immediately.

I added a status to the idempotency record.

An operation can now be:

processing

or:

completed

When the first request arrives:

Check key
    |
    v
Not found
    |
    v
Store PROCESSING
    |
    v
Perform payment

Now imagine another request with the same key arrives while the payment is still being processed.

It checks the store and sees:

PROCESSING

so it doesn’t start another payment.

Once the first operation finishes, the idempotency record changes to:

COMPLETED

and stores the result.

The lifecycle becomes:

New request
    |
    v
PROCESSING
    |
    v
Payment succeeds
    |
    v
COMPLETED

In my example, if another request arrives while the first one is still processing, I return 409 Conflict.

Another implementation might make that request wait for the first operation to finish and then return the same response.

The exact behavior can differ.

The important part is that the second request does not perform the side effect again.

What if the payment fails?

Adding the processing state creates another problem.

Suppose we do this:

payment-123 -> PROCESSING

and then the payment operation throws an error.

If we leave that record there, every future retry will see:

PROCESSING

and assume another request is still working on it.

The key would effectively be stuck forever.

For this example, I remove the idempotency record when processing fails.

That means the client can retry the operation.

Again, this is a design decision.

A production API might choose to remember certain failures, allow retries for others, or have more states.

The thing I took away from this is that idempotency isn’t just:

Save a response in a Map.

You also have to think about the lifecycle of the operation and what should happen when each part fails.

Testing the guarantees

I used Vitest and Supertest to make sure the implementation actually guarantees the behavior I wanted.

The main cases I tested were:

Same key + same payload
-> only one payment is created

Same key + different payload
-> request is rejected

Different keys
-> separate payments are created

Concurrent requests with the same key
-> only one payment is created

The concurrency test was the most interesting one.

Instead of sending one request, waiting for it to finish, and then sending another, both requests are started together:

const [firstResponse, secondResponse] = await Promise.all([
  firstRequest,
  secondRequest,
]);

The important assertion isn’t just what HTTP status codes come back.

The test also verifies that only one payment exists.

That is ultimately what we care about.

The thing idempotency is protecting is the side effect.

What this example doesn’t solve

The implementation in this project is deliberately small.

The idempotency store is just an in-memory JavaScript Map.

That works for demonstrating the idea, but it isn’t enough for a real distributed application.

Imagine the application is running on two servers:

              Load Balancer
               /         \
              /           \
         Server A       Server B
           Map A          Map B

The first request could reach Server A while the retry reaches Server B.

Server B has its own Map, so it has no idea Server A is already processing that idempotency key.

Both servers could create the payment.

A production implementation would normally use shared storage such as Redis or a database.

There is also another important detail: claiming the key needs to be atomic.

Two application instances shouldn’t both be able to check:

Does this key exist?

see that it doesn’t, and then both claim it.

There are several other things I deliberately left outside this example:

• Expiring old idempotency records with a TTL
• Shared storage across application instances
• Atomic key claiming
• Deciding which failures should be stored
• Deciding how concurrent callers should behave
• Scoping keys to users, accounts, or endpoints
• More robust request fingerprinting
• Recovering if a server crashes while an operation is still marked as processing

The purpose of this example wasn’t to build a production payment system.

I wanted something small enough that I could understand why each part of an idempotency implementation exists.

What I took away from building it

Before working through this example, I mostly thought of idempotency as:

If I receive the same idempotency key twice, don’t do the thing twice.

That’s still the central idea, but there is more hidden inside that sentence than I originally thought.

• Verify that a repeated key represents the same operation.
• Account for two requests arriving at the same time.
• Decide how callers should be handled while an operation is processing.
• Decide which failures should be stored and returned.
• Use shared storage and an atomic way to claim operations when multiple application instances are involved.

Building the broken version first made these problems much easier for me to understand than starting with a finished implementation.

That is the approach I want to take with more of the things I write about here: start with something I understand at a surface level, build a small example, find where my understanding or implementation breaks, and use that process to understand why the real solution looks the way it does.

Trying the example

The source code is available here:

GitHub — Idempotency Demo

To start with the payment API before idempotency is implemented, use the starting-point branch:

git switch starting-point

That contains the basic Express payment API before idempotency is implemented.

The completed version is on the master branch:

git switch master

The repository also contains tagged snapshots for the starting and completed versions.

If you’re learning idempotency as well, I recommend starting with the broken endpoint before looking at the completed implementation.

Seeing exactly where each version failed was what made the concept click for me.