Building an order API that never charges twice
Why order APIs go wrong
An order request does three things that can each fail on their own: it takes money, it asks a supplier for something, and it records the result. Networks time out between any two of those. When a client sees a timeout it cannot tell whether the order happened, so it retries — and a naive server happily creates a second order and charges for it.
Most of the engineering in a reliable order API is about making that retry harmless.
How do you make a retry return the same order?
The standard answer is an idempotency key. The client generates a unique reference before its first attempt, sends it with the request and reuses it on every retry. The server stores the order against that reference, and when it sees the reference again it returns the original order, flagged as a replay, instead of creating a new one.
Two details trip people up. The first is concurrency: if the original request and the retry arrive at the same moment, both can pass the 'have I seen this?' check. Serialise requests that share a reference — in PostgreSQL an advisory lock keyed on the reference does it cheaply. The second is on the client: it has to persist the reference before sending, not generate a new one inside the retry loop.
Why hold the money before calling the supplier?
In a prepaid model the order should reserve the cost in the wallet first, call the supplier second, and then settle the hold on success or release it on failure. That ordering means two simultaneous orders cannot both spend the same balance, and a rejected order never leaves money stuck.
Orders for several units need a rule too. We order units one at a time; if a later unit fails, the response says the order was only partly fulfilled and only the units that were delivered are charged. All-or-nothing sounds cleaner, but it means undoing things the supplier has already done.
Give an unclear outcome its own state
The worst case is a supplier call that times out: the order may or may not exist on their side. Guessing in either direction is wrong — assume failure and you may deliver twice, assume success and you may deliver nothing.
Record the order before you call the supplier, mark a timed-out call as ambiguous, reconcile it later with a read-only lookup, and alert a person if it stays unresolved. In our MobiSIM API an ambiguous order is held exactly that way, so a customer is never bought twice.
Webhooks: tell the client what happened, safely
Some orders finish after the response — a plan is issued a moment later — so clients need to be told. Send a signed event: compute an HMAC over a timestamp plus the raw body with a secret only you and the client know, and have the client verify it with a constant-time comparison and reject events whose timestamp is too old, which stops replays. We use a five-minute tolerance.
Delivery should retry with backoff — ours keeps trying for about two days — and you should say plainly that it is at-least-once and can arrive out of order. That puts the work where it belongs: clients de-duplicate on the event ID and treat the order endpoint as the source of truth, using the webhook as a nudge to go and look.
A test mode is part of the API
Give integrators a way to exercise the whole flow without risk: an endpoint that returns the real response shape with a placeholder result, buys nothing and charges nothing. It shortens integrations, and it means a customer's first real order is not also their first test.
A short checklist
If you are building or reviewing an order API, these are the questions we would ask:
- Is there an idempotency reference on every purchase, serialised per reference?
- Is the money held before the supplier call, then settled or released afterwards?
- Is partial fulfilment a first-class result?
- Are ambiguous outcomes recorded, reconciled and escalated to a person?
- Are webhooks signed, timestamped, retried and documented as at-least-once?
- Is there a test mode that costs nothing?