> ## Content Index > Fetch the complete content index at: https://stalin-pereira.com/llms.txt > Use this file to discover other available public pages before exploring further. # Designing Idempotency Keys for Payment APIs - URL: https://stalin-pereira.com/designing-idempotency-keys-for-payment-apis/ - Published: 2025-12-18T04:30:00.000Z - Updated: 2026-10-16T19:23:20.000Z - Description: A lost response plus an automatic retry equals a double charge. How to design idempotency keys properly: client rules, a Postgres schema, atomic key claims, concurrent retries, crash recovery around external calls, and the edge cases. - Author: Stalin Pereira - Tags: Backend A user taps "Pay $50". The request reaches your server, the card gets charged, and the response is lost on a flaky mobile connection. The app sees a timeout and retries. Your server, seeing what looks like a brand new request, charges the card again. The user now has two $50 charges and a reason to never use your product again. This failure is not exotic. Any time a client can't tell whether a request succeeded, it has to choose between retrying (risking a duplicate) and giving up (risking a lost payment). Without idempotency keys you can't do both safely. If a payment API can't handle a retry, it isn't ready for production. This article covers how to design idempotency keys properly: what the client sends, how the server stores and checks keys, how to deal with concurrent retries and crashes halfway through a request, and where the edge cases hide. The examples use PostgreSQL and Python, but the design carries over to any stack. ## What idempotency actually means here An operation is idempotent if doing it several times has the same effect as doing it once. HTTP already defines `GET`, `PUT` and `DELETE` as idempotent. `POST` is not, and "create a charge" is about as non-idempotent as an operation gets. An idempotency key turns a `POST` into something the client can safely retry. The client generates a unique value, usually a UUID, for each *intent* ("charge this order once") and sends it with every attempt at that intent: ```http POST /v1/payments HTTP/1.1 Host: api.example.com Authorization: Bearer sk_live_... Content-Type: application/json Idempotency-Key: 8e03978e-40d5-43e8-bc93-6894a57f9324 {"order_id": "ord_1042", "amount": 5000, "currency": "USD"} ``` The server's contract is simple to state. The first request with a given key runs normally and its result is recorded. Any later request with the same key gets that recorded result back without running the operation again. Plenty of APIs work this way. [Stripe's idempotent requests](https://docs.stripe.com/api/idempotent%5Frequests?ref=stalin-pereira.com) are the reference most teams copy, and the IETF HTTPAPI working group has a draft standardising the header: [The Idempotency-Key HTTP Header Field](https://datatracker.ietf.org/doc/draft-ietf-httpapi-idempotency-key-header/?ref=stalin-pereira.com). Following both closely is cheaper than inventing your own semantics. ## The client side: one key per intent A lot of idempotency bugs live on the client, not the server. The rules: - **Generate the key once per user intent, not once per HTTP attempt.** If your retry wrapper creates a new UUID on each attempt, you have retries without idempotency. - **Persist the key before the first attempt.** If the app is killed after sending but before getting a response, the retry after restart has to reuse the same key. Store it with the pending order in local storage or your own database. - **Change the key only when the intent changes.** If the user edits the amount, that's a new payment and gets a new key. If the server returned a definite failure (card declined), retrying with the same key just replays the decline. A new attempt needs a new key. - **Back off with jitter.** Thousands of clients retrying in lockstep after an outage make it worse. Use exponential backoff with random jitter. A minimal retrying client in Python with `httpx`: ```python import random import time import uuid import httpx def create_payment(client: httpx.Client, order: dict, idempotency_key: str, max_attempts: int = 5) -> httpx.Response: # idempotency_key is created and stored by the caller, once per order for attempt in range(max_attempts): try: resp = client.post( "/v1/payments", json=order, headers={"Idempotency-Key": idempotency_key}, timeout=10.0, ) except httpx.TransportError: pass # timeout or connection failure: outcome unknown, safe to retry else: # 409 = an earlier attempt with this key is still running if resp.status_code not in (409, 429) and resp.status_code < 500: return resp time.sleep(min(30.0, 0.5 * 2 ** attempt) * random.uniform(0.5, 1.0)) raise RuntimeError(f"payment for key {idempotency_key} did not complete") # when the user confirms the order: # key = str(uuid.uuid4()); save key alongside the pending order; then call create_payment ``` Note that the client retries on transport errors, 409, 429 and 5xx, but never on other 4xx responses. A 400 or 422 will come back the same every time. ## The server side: a table, not a cache Redis with a TTL is the obvious place to put idempotency keys, and it's a reasonable choice for many APIs. For payments, put them in the same transactional database as the payment records instead. The reason is atomicity. The point where you record "this key produced payment `pay_123`" should commit in the same transaction as the payment row itself. If they live in different systems, there's a window where one is written and the other isn't, and a crash in that window gives you a duplicate or a lost result. Redis is also allowed to lose data in ways a payments ledger is not: eviction under memory pressure, or a failover to a replica that hadn't caught up. Redis is fine as a fast-path lock in front of the database. It shouldn't be the source of truth. A PostgreSQL schema that works well: ```sql CREATE TABLE idempotency_keys ( account_id bigint NOT NULL, idempotency_key text NOT NULL CHECK (length(idempotency_key) <= 255), request_method text NOT NULL, request_path text NOT NULL, request_hash bytea NOT NULL, -- SHA-256 of the canonicalised body status text NOT NULL DEFAULT 'started' CHECK (status IN ('started', 'completed')), locked_at timestamptz, response_code int, response_body jsonb, created_at timestamptz NOT NULL DEFAULT now(), PRIMARY KEY (account_id, idempotency_key) ); CREATE INDEX idempotency_keys_created_at_idx ON idempotency_keys (created_at); ``` A few of these columns are doing more than they look: - **`account_id` in the primary key** scopes keys per tenant. Two merchants who both happen to send `order-1` as a key must not collide, and one tenant must never be able to replay another tenant's response. - **`request_hash`** lets you detect a client reusing a key with a different payload, which is almost always a client bug and must not silently return the old result. - **`locked_at`** marks a request in flight, so a concurrent retry knows to back off instead of charging in parallel. ## The request flow Every request with an idempotency key goes through the same steps: 1. Try to insert a new row for `(account_id, key)`. If the insert succeeds, this request owns the key. 2. If the row already exists, compare request hashes. Different hash: reject with 422\. Same hash and completed: return the stored response. Same hash and still locked: return 409 so the client retries later. 3. If you own the key, do the work, then store the response and mark the row completed. The insert-first approach matters. A "check, then insert" sequence has a race: two concurrent requests both see no row and both proceed. `INSERT ... ON CONFLICT DO NOTHING` lets the primary key constraint make that decision atomically. ```python import hashlib import json import psycopg from psycopg.types.json import Jsonb class IdempotencyConflict(Exception): def __init__(self, status: int, message: str): self.status, self.message = status, message def request_fingerprint(body: dict) -> bytes: canonical = json.dumps(body, sort_keys=True, separators=(",", ":")) return hashlib.sha256(canonical.encode()).digest() def begin(conn: psycopg.Connection, account_id: int, key: str, method: str, path: str, body: dict): """Returns None if this request owns the key, or a stored (code, body) to replay.""" fp = request_fingerprint(body) with conn.transaction(): inserted = conn.execute( """ INSERT INTO idempotency_keys (account_id, idempotency_key, request_method, request_path, request_hash, locked_at) VALUES (%s, %s, %s, %s, %s, now()) ON CONFLICT (account_id, idempotency_key) DO NOTHING RETURNING 1 """, (account_id, key, method, path, fp), ).fetchone() if inserted: return None row = conn.execute( """ SELECT request_method, request_path, request_hash, status, response_code, response_body FROM idempotency_keys WHERE account_id = %s AND idempotency_key = %s FOR UPDATE """, (account_id, key), ).fetchone() req_method, req_path, req_hash, status, code, resp_body = row if (req_method, req_path, bytes(req_hash)) != (method, path, fp): raise IdempotencyConflict(422, "Idempotency-Key reused with a different request") if status == "completed": return code, resp_body raise IdempotencyConflict(409, "A request with this Idempotency-Key is in progress") def complete(conn: psycopg.Connection, account_id: int, key: str, code: int, body: dict) -> None: # Call this inside the same transaction that writes the payment row. conn.execute( """ UPDATE idempotency_keys SET status = 'completed', locked_at = NULL, response_code = %s, response_body = %s WHERE account_id = %s AND idempotency_key = %s """, (code, Jsonb(body), account_id, key), ) ``` In the handler, `begin()` runs first. If it returns a stored response, send it back as-is. If it raises, map the exception to the HTTP status. Otherwise run the payment logic and call `complete()` in the same transaction that inserts the payment record. The status codes follow the IETF draft: 400 if the endpoint requires a key and none was sent, 422 for a key reused with a different payload, and 409 while the original request is still being processed. It's also worth adding a response header such as `Idempotent-Replayed: true` on replays, so clients and support engineers can tell a replay from a fresh result. ## The hard part: the external call in the middle The flow above assumes the work happens inside one database transaction. Payments almost never do. Somewhere in the middle you call a card processor over the network, and you cannot roll that back. Consider what happens if your process crashes after the processor charged the card but before you committed. The idempotency row says `started`, nothing says `completed`, and the client's retry gets a 409 forever, or worse, if you clear stale locks naively, runs the charge a second time. Two techniques together handle this. **Propagate idempotency downstream.** When you call the processor, pass an idempotency key of your own derived deterministically from the client's key, for example `sha256(account_id + ":" + key + ":authorize")`. If your service retries the processor call after a crash, the processor deduplicates it. This only works if the downstream system supports idempotency. Major payment processors generally do, but check before you rely on it. **Break the request into recoverable phases.** Instead of a binary started/completed state, record progress as the request moves through steps that each commit atomically: `started`, `payment_created`, `charge_submitted`, `completed`. A retry that finds an expired lock picks up from the last recorded phase rather than starting over. Brandur Leach's [Implementing Stripe-like Idempotency Keys in Postgres](https://brandur.org/idempotency-keys?ref=stalin-pereira.com) describes this "recovery point" design in detail and is worth reading in full before you build one. Pair this with a lock timeout. If `locked_at` is older than your longest plausible request duration, a retry may take over the key and resume from the recorded phase. Choose the timeout carefully: too short and a slow but alive request gets taken over while it's still running. ## Decisions you'll need to make ### Which outcomes get stored? Store any result produced after the operation started executing, including failures like a card decline. A retry of a declined charge should return the same decline, not try the card again. Don't store results from requests rejected before execution, such as failed validation or authentication. Stripe's v1 API documents the same rule: results are saved only once endpoint execution begins, and a retry gets the saved response, even a `500`. Server errors are the hard case. If a 500 happened before any side effect, leaving the key retryable is friendlier. If it happened after the processor call, you need the recovery logic above, not a cached 500. ### How long do keys live? Keys must outlive the client's maximum retry window, including retries from apps that were offline for a while. Stripe's v1 API may prune keys once they're at least 24 hours old, which is a sensible default. Delete expired rows with a scheduled job using the `created_at` index. If you need longer-lived protection against duplicate orders, add a separate uniqueness constraint on a business identifier such as `(account_id, order_id)`. Don't stretch key retention to cover it. ### What's a valid key? Accept opaque strings up to a bounded length (255 characters is common) and recommend UUIDv4 or another high-entropy value. Don't let the server generate keys. The whole point is that the client knows the key before it learns the outcome. ## Where idempotency keys stop helping Idempotency keys handle one specific problem: a retry of the same request from the same client. They don't cover: - **Two different requests for the same business intent.** A user double-clicking a button that isn't disabled, with the frontend generating two keys, produces two payments. Use business-level uniqueness (one payment per order) for that. - **Duplicate events flowing through your own system.** Once the payment is accepted, downstream processing via queues and event streams has its own duplicate problem. Consumers need their own deduplication, covered in [Handling Duplicate Messages in Kafka Consumers](https://stalin-pereira.com/handling-duplicate-messages-in-kafka-consumers/). - **Naturally idempotent operations.** "Set the shipping address to X" via `PUT` is already safe to repeat. Adding keys there is overhead with no benefit. They're also not free. Every keyed request costs at least one extra write, response bodies take up storage, and the crash-recovery logic is real code that needs testing. For an internal read-heavy service, that's probably not worth it. For anything that moves money, sends messages or creates resources a user will be billed for, it is. ## Checklist - Clients generate one key per intent, persist it before the first attempt, and reuse it on every retry. - Clients retry on timeouts, 409, 429 and 5xx with exponential backoff and jitter. Never on other 4xx. - Keys are scoped per account in the primary key. - The server claims a key with an atomic insert, not check-then-insert. - The request fingerprint is stored and compared; mismatches return 422. - In-flight duplicates get 409; completed duplicates get the stored response with a replay header. - The completed response commits in the same transaction as the business write. - Downstream calls carry their own deterministic idempotency keys. - Multi-step requests record recovery points and have a lock timeout. - Keys expire after a window longer than any client's retry horizon, with a cleanup job. - You've tested the ugly paths: concurrent retries, a crash after the processor call, and a key reused with a different body.