Learn

The app retried after a timeout, and the customer paid twice

28 October 2026 · System design: one trade-off

You have drawn a checkout: the app calls your payment service, which calls the card provider. The interviewer asks one thing. The app waits three seconds, gets no answer, and sends the same request again. The provider took four seconds, and both requests reached it. How many times was the card charged?

The requirement

A request that is sent again, for any reason, must not move money again. The caller may retry after a timeout, a dropped connection or a crash, and it cannot tell which of those happened. So the server has to recognise a repeat and answer it without charging.

That is narrower than "exactly once delivery". You are not promising that a message arrives once. You are promising that the effect happens once, however many times the request arrives. Saying that difference out loud early tells the interviewer you know which promise is cheap and which is not.

Two designs and what each one costs

Option A: an idempotency key per attempt. The caller makes up a random key before the first send and puts the same key on every retry. The server stores the key with the outcome and replays the stored answer when the key comes back.

• What you get: one rule on one endpoint, and it works for any caller, including other companies calling a public API.
• What you pay: a table of keys with an expiry, a decision about how long to keep them, and a third state between "new" and "done".

def pay(store, gateway, key, order_id, amount):
    seen = store.insert_if_absent(key, {"state": "pending", "order": order_id})
    if seen is not None:                     # this key has been used before
        if seen["order"] != order_id:
            return {"status": 422}           # same key, different request
        if seen["state"] == "done":
            return seen["result"]            # replay the first answer
        return {"status": 409}               # first attempt still running
    result = gateway.charge(order_id, amount, idempotency_key=key)
    store.update(key, {"state": "done", "order": order_id, "result": result})
    return result

Run it on the interviewer's case. The first request inserts the key as pending and starts the charge. The retry arrives one second later, finds the key pending and gets a 409, so it waits and asks again. Once the first call finishes, the retry gets the stored result. The provider was called once. The insert has to be a single atomic step, such as a unique index, or two retries arriving together both see "absent".

Option B: the order carries its own state. The app first creates an order, which charges nothing and returns an id. Paying is then a change from created to charging on that id, done with one conditional update: set the state only where it is still created. A retry finds the state already moved and reads the outcome instead of charging.

• What you get: no separate key table. The order row already exists, and its state is the record of what happened.
• What you pay: two calls instead of one, and the caller has to keep the order id between them.

What goes wrong, and the input that shows it

Option A breaks when your own server dies between the charge and the update. The key stays pending forever, and every retry gets 409. You need an expiry on pending keys, and you need to pass the same key down to the provider, so that running the charge again after the expiry is safe there too.

Option B breaks one step earlier. The app is killed after the create call but before it saved the id. On restart it creates a second order. Nobody is charged twice, but the customer now has two orders, one of them unpaid, and someone has to clean them up.

What to say out loud

Start with the deciding question: "Is there already a record of this purchase on our side before money moves, and are the callers only our own app, or outside developers too?" Then commit: "If an order exists first and only our app calls us, I make paying a one-way state change on that order. If anyone can call the endpoint with nothing created beforehand, I require an idempotency key, store it with a pending state, and pass it to the provider."

The exercise

1. Pick an action with a side effect that hurts when doubled: sending a text, booking a seat, issuing a refund.
2. Write option A and option B for it in one line each, with what each needs stored.
3. Write the two inputs that break them: a crash between the side effect and the record, and a client that loses the id it was given.
4. Say the deciding question aloud, then your choice for two different answers to it. Aim for under a minute.

Where this fits in Gadfly

System design is one of Gadfly's four rooms, next to algorithms, behavioral and code review. The design runs on a shared board, guided or as a mock, one trade-off at a time. In guided mode the tutor asks the next question, such as what happens when the server dies between the charge and the record, and does not draw the answer. In a mock it behaves like an interviewer, with no coaching during the round and a debrief at the end that quotes what was said, not a grade.