Idempotency keys: check-then-insert double-charged 216 times

The most common idempotency design, "look up the key, and if it is missing do the work and store it", double-charged 216 times in our lab when 1,000 payments were each retried five times at once. A single atomic claim on the key brought the duplicates to zero. A crash between the charge and the "done" mark then brought back 100 duplicates in three of the five designs we tried.

This post gives the working pattern, the numbers behind it, and the one extra rule that fixes the crash case. Everything below is code you can paste into a Node API.

What is an idempotency key, and why do retries need one?

An idempotency key is a unique token a client sends with a request so that the server can recognise a retry and return the first result instead of running the action again. Idempotency is the property that doing an operation twice has the same effect as doing it once.

Retries are not optional. Mobile networks drop responses, load balancers time out, and a client that never saw the answer to a payment will send it again. Without a key the server cannot tell a retry from a second purchase.

Stripe's API reference on idempotent requests describes the widely copied contract. The client generates the key, Stripe recommends a V4 UUID, and keys can be up to 255 characters long. For API v1, keys can be removed after they are at least 24 hours old. The idempotency layer also compares the incoming parameters to those of the original request and errors if they differ.

A standard is on the way, but it is not finished. The IETF draft on the Idempotency-Key HTTP header says the header can make non-idempotent methods such as POST or PATCH fault-tolerant. Its datatracker page lists the draft as an expired Internet-Draft, last updated on April 18, 2026. Treat the header name as a convention, not a standard.

What those pages do not show is how each way of storing the key behaves under a retry storm or a crash. We tested that.

How we tested five designs

The test setup and methodology are below, so you can check them against your own stack.

The test setup was PostgreSQL 18.1 and Redis 8.4.0 on a Mac laptop, with Node.js 22 using the pg and ioredis drivers. Each "payment" was a row inserted into a charges table, and it is the side effect we must never repeat.

We built five handlers:

1. None. No key at all, as a baseline.
2. Check-then-act. SELECT the key. If it is missing, charge, then INSERT the key.
3. Postgres claim. INSERT ... ON CONFLICT DO NOTHING a pending row first. Only the request that inserted it charges, then marks it done.
4. Postgres transaction. Claim the key and charge inside one transaction, so both commit or neither does.
5. Redis SET NX. Claim with SET key pending PX 60000 NX, charge, then store done.

For the duplicate storm we sent 1,000 distinct keys, each five times, in shuffled order from 100 concurrent workers. The charge took 20 ms, which gives the retries time to overlap. The number we count is duplicateCharges: rows in charges minus distinct keys.

For the crash test we killed the worker after the charge and before the completion mark on 100 of 1,000 keys. The client retried after 120 ms, and a pending key counted as stale after 100 ms.

Results: does check-then-insert really fail?

Yes. The table shows what happened to 5,000 requests that should have produced exactly 1,000 charges.

Design	Charges made	Duplicate charges	Requests per second
No key	5,000	4,000	3,881
Check-then-act	1,216	216	10,255
Postgres claim	1,000	0	6,656
Postgres transaction	1,000	0	3,392
Redis SET NX	1,000	0	12,637

Check-then-act looks fast and looks safe, which is why it ships. Its speed is the giveaway. It was the fastest Postgres design because it skipped the coordination. A second run gave 165 duplicates instead of 216, so the count is noisy, but it was never zero.

A race condition is a bug where the outcome depends on the timing of concurrent operations. Here, two retries both run the SELECT, both see no key, and both charge. Widen the window with a slow payment processor and the duplicates get worse, not better.

Your tests will not catch this. A unit test sends one request. A load test with unique keys sends no retries. The failure needs duplicates that overlap in time, which is exactly what a client timeout produces in production.

Why does a unique constraint fix the race?

Because the database, not your code, decides who wins. Both the Postgres claim and Redis SET NX are single atomic operations that succeed for exactly one caller.

The PostgreSQL INSERT documentation says ON CONFLICT DO NOTHING avoids raising an error when a row violates a unique constraint. With the key as the primary key, one insert returns a row and the rest return none. The one that got a row owns the work.

Redis does the same job with a different command. The Redis SET command reference documents the NX option, which sets the key only if it does not exist, and PX, which sets an expiry in milliseconds. One client gets OK and the others get nothing.

Here is the Postgres claim as a Node handler. It is the design we recommend as a default.

// schema: create table idem(key text primary key, status text not null,
//   response jsonb, created timestamptz not null default now());
async function withIdempotency(pool, key, work) {
  const claim = await pool.query(
    "insert into idem(key, status) values ($1, 'pending') on conflict do nothing returning key",
    [key]
  );
  if (claim.rowCount === 1) {
    const response = await work(key);          // the one and only execution
    await pool.query(
      "update idem set status = 'done', response = $2 where key = $1",
      [key, response]
    );
    return { replayed: false, response };
  }
  // someone else owns this key: wait for them, then replay their result
  for (let i = 0; i < 200; i++) {
    const row = await pool.query("select status, response from idem where key = $1", [key]);
    if (row.rows[0].status === "done") return { replayed: true, response: row.rows[0].response };
    await new Promise((r) => setTimeout(r, 25));
  }
  throw Object.assign(new Error("request still in progress"), { status: 409 });
}

Two details matter. The loser polls until the winner finishes, and gives up with a 409 after five seconds instead of hanging a connection forever. And the claim row exists before the work starts, which is what closes the race.

Which is faster, Postgres or Redis?

Redis, by about three times in our run. With 4,000 unique keys, 64 concurrent workers and no artificial delay, the median of three runs was:

Design	Requests per second (median of 3)
Redis SET NX	8,837
Postgres claim	2,993
Postgres transaction	2,644

The charge itself still went to Postgres in all three, so the gap is the cost of the key store alone, plus one extra round trip for the completion mark. A second correct design running at 2,000 to 3,000 requests per second is plenty for most payment endpoints. Few APIs take more than that many state-changing requests per second.

The speed difference is therefore not the deciding factor. Durability and atomicity are, and the crash test is where they separate.

What happens when the worker dies mid-request?

This is where most guides stop, and where the real money is lost. The server charged the card, then the process crashed before it recorded done. The key is stuck at pending. The client retries. What now?

You have two choices. Return an error forever, which strands a customer who was charged. Or treat a stale pending as abandoned and run the work again, which is what we tested after 100 ms.

Design	Charges made	Duplicates after 100 crashes
Postgres claim, takeover when stale	1,100	100
Postgres transaction, local write	1,000	0
Postgres transaction, external call	1,100	100
Redis SET NX, expiry as takeover	1,100	100
Postgres transaction, unique key on the processor	1,000	0
Postgres claim, unique key on the processor	1,000	0

Read the second row first. When the charge is a row in the same database, one transaction makes the key and the charge commit together. A crash rolls both back, the retry starts clean, and the duplicates are zero. That is the strongest argument for keeping the side effect in Postgres whenever you can.

The third row is the trap. Wrap an external call in a transaction and the transaction guards nothing. The processor committed its own state before our database rolled back. The retry found no key and charged again, 100 times.

Redis fails the same way for a different reason. Its key expires or goes stale, and the next request takes over. Redis is a fine lock. It cannot know whether the side effect already happened.

The extra rule: put the key on the side effect

Every duplicate in the crash test vanished with one change. We passed the same key to the processor and gave its table a unique constraint, so a repeated insert did nothing.

create table processor (
  id bigserial primary key,
  key text unique          -- the processor rejects a repeated key
);
-- the call becomes:
insert into processor(key) values ($1) on conflict do nothing;

With a real payment provider you do not write that table. You forward your key in the provider's idempotency header, which Stripe supports as described above. The principle is the same: the system that performs the irreversible action must be the one that refuses to repeat it. Your key store only decides who gets to ask.

This also changes how you read the Redis versus Postgres question. If the side effect is idempotent by itself, either store is safe and Redis is faster. If the side effect is not, neither store is safe on its own, and the choice does not matter.

A complete, safe handler

Combine the pieces. Claim the key, forward it downstream, and make the stale-pending takeover safe because the downstream call is repeatable.

app.post("/payments", async (req, res) => {
  const key = req.get("Idempotency-Key");
  if (!key || key.length > 255) return res.status(400).json({ error: "Idempotency-Key required" });

  try {
    const out = await withIdempotency(pool, key, async () => {
      // forward the same key, so a repeat after a crash is refused downstream
      return processor.charge({ amount: req.body.amount, idempotencyKey: key });
    });
    res.set("Idempotent-Replayed", String(out.replayed)).json(out.response);
  } catch (e) {
    res.status(e.status ?? 500).json({ error: e.message });
  }
});

Three more habits finish the job:

1. Hash the request body and store it with the key. If the same key arrives with different parameters, return a 422 instead of replaying the wrong result. Stripe does this comparison, as the API reference notes.
2. Expire keys on a schedule. A daily delete from idem where created < now() - interval '24 hours' mirrors Stripe's window and keeps the table small.
3. Use random keys. A V4 UUID works. If you want time-ordered keys for index locality, we measured that trade in UUIDv7 or UUIDv4 keys in Postgres: 11M rows, 22x slower.

Where does the outbox pattern fit?

When one request must both write a row and publish an event, a transaction cannot span the database and the message broker. The outbox pattern writes the event to an outbox table in the same transaction as the business row. A separate relay then publishes it.

The relay can crash after publishing and before marking the row sent, which is the same crash window as above. Consumers therefore need their own idempotency, and the event id is the natural key. The pattern moves the problem to a place where the same fix applies: a unique key on the consumer's side effect.

Which design should you pick?

Pick by where the side effect lives.

• The side effect is a row in your own Postgres. Use one transaction that inserts the key and the work together. It scored 0 duplicates, even under a crash, with no extra rules. Throughput was 2,644 requests per second in our run.
• The side effect is an external API that supports idempotency keys. Use the Postgres claim, and forward your key. It scored 0 duplicates in both the storm and the crash test, at 2,993 requests per second.
• You need very high request rates and the side effect is repeatable. Use Redis SET NX. It was the fastest at 8,837 requests per second, but only combine it with a downstream key.
• You are tempted by check-then-act. Do not ship it. It is the design that made 216 duplicates.

If you also protect these endpoints with throttling, the burst behaviour matters for retries. We compared the algorithms in Rate limiter algorithms compared: the 2x burst most miss.

What this lab did not cover

Honesty about limits makes the numbers more useful.

The charge in our lab was a local insert with a 20 ms delay, not a real card network. A real processor adds seconds of latency, which widens the race and favours the claim design more. We ran on one machine, so network partitions and Redis failover were not tested. A replica promoted without the latest SET would drop a claim, and that case needs its own test. We also did not test Postgres under heavy lock contention on one hot key, where waiting retries queue behind the winner.

Your hardware will produce different throughput. The duplicate counts are the portable part, because they come from logic, not speed.

The bottom line: the key store decides who asks, the side effect decides who wins

Check-then-insert made 216 duplicate charges in a storm of 5,000 requests. An atomic claim made none. A crash then showed that a perfect key store still cannot protect a side effect it does not control, and 100 customers were charged twice in three designs until the processor itself refused a repeated key.

Claim the key atomically, keep the side effect in the same transaction when it is a local row, and forward the key to anything outside your database. Then test it with duplicates that overlap and a worker you kill on purpose.