Payment systems are among the most unforgiving environments in software engineering. A user clicks “Pay Now” and within 300 milliseconds, that request crosses your API gateway, bounces through an idempotency service, orchestrates calls to a third-party payment provider like Stripe or Braintree, writes to your ledger, and publishes an event to trigger downstream fulfillment. At any point in that chain, a network timeout, a pod restart, or a transient 503 from the payment provider can leave the transaction in an ambiguous state - the charge may or may not have been applied, but the API never returned a definitive response.

Think of a payment processor like a postal certified-mail system. Every letter gets a unique tracking number before it leaves your hands. If the delivery truck breaks down midway, the postal service doesn’t simply send another identical letter without checking whether the first arrived. It uses the tracking number to determine whether delivery was completed before dispatching a replacement. Idempotency keys are your tracking numbers: they let any component in the system - client, orchestrator, provider adapter - answer the question “did this specific operation already succeed?” before taking any irreversible action.

The scale of the problem amplifies the complexity. A mid-sized e-commerce platform processes 50,000 transactions per hour at peak, spread across 200 concurrent API nodes. Each node independently retries timed-out requests with exponential backoff. Without a coordinated idempotency layer, a payment that timed out at the provider returns 408 to the client, the client retries three times, and you submit the same charge four times. Stripe returns success for the first and duplicate-charge errors for the rest - if you are lucky. If the first returned a timeout too, all four attempts look like new charges.

The ledger consistency dimension adds another layer. Crediting a user’s wallet, debiting their bank account, and recording the transaction in your double-entry ledger are three separate writes across potentially three separate services or databases. If the wallet credit succeeds, the bank debit succeeds, but the ledger write fails due to a deadlock, your books are out of balance. A reconciliation job will catch this in the morning, but in the meantime your system reports a balance that doesn’t match reality. We need to solve for idempotent retries, distributed transaction atomicity, and ledger consistency simultaneously.

Requirements and Constraints

Functional Requirements

• Accept payment requests with a client-supplied idempotency key and return identical responses for duplicate requests
• Support payment providers Stripe, Braintree, and PayPal with a unified adapter interface
• Maintain a double-entry ledger recording every debit and credit with transaction-level atomicity
• Retry failed payment attempts with exponential backoff without issuing duplicate charges
• Expose payment status endpoint returning the current state and all transition history
• Support full and partial refunds that are themselves idempotent
• Run a reconciliation job that detects ledger imbalances and provider-side discrepancies within 24 hours

Non-Functional Requirements

• Process 50,000 transactions per hour sustained, with 200,000 peak for flash sale events
• P99 payment latency under 2 seconds end-to-end including provider round-trip
• Idempotency store lookups must complete in under 5ms at P99
• Ledger writes must be durable - zero data loss under single-node failures
• Payment state changes must be auditable for 7 years (regulatory requirement)
• System must remain operational during a single availability zone outage

Constraints and Assumptions

• Payment providers implement their own idempotency keys on their side; we pass ours through
• Idempotency keys expire after 24 hours; re-using an expired key creates a new payment
• Maximum concurrent in-flight payments per user: 5 (fraud prevention constraint)
• Double-entry ledger is authoritative; provider records are reconciled against it
• All monetary amounts stored as integers in the smallest currency unit (paise for INR, cents for USD)

High-Level Architecture

Six components form the backbone of the system: a Client that supplies idempotency keys with every request, an API Gateway that handles authentication and rate limiting, an Idempotency Service that deduplicates requests before any downstream work begins, a Payment Orchestrator that drives the saga across provider and ledger, a set of Payment Provider Adapters (Stripe, Braintree, PayPal), and a Storage Layer consisting of the Ledger DB, idempotency key store, and Event Bus.

A new payment request arrives at the API Gateway carrying an Idempotency-Key header. The gateway passes it to the Idempotency Service, which performs a Redis lookup using the hashed key. On a cache miss, it creates a new idempotency record with status IN_PROGRESS and a distributed lock, then allows the request to proceed to the Payment Orchestrator. The Orchestrator drives the saga: reserve the funds in the user’s account, call the selected payment provider adapter, write credit and debit entries to the Ledger DB in a single transaction, and emit a payment.succeeded event to the Event Bus. On saga completion, the Idempotency Service updates its record with the final response so future duplicates receive an instant cached reply.

On a cache hit, the Idempotency Service inspects the stored record’s status. If COMPLETED, it returns the cached response immediately without touching the Orchestrator. If IN_PROGRESS, it returns 202 Accepted and the client polls the status endpoint. This prevents the thundering-herd scenario where a retry storm during a slow provider response causes hundreds of duplicate saga executions for the same payment.

The Idempotency Layer