EasyEnv

Webhook Idempotency Interview: The Backend Exercise Where the Webhook Arrives Twice

November 19, 2026·7 min read·By EasyEnv
Webhook Idempotency Interview: The Backend Exercise Where the Webhook Arrives Twice

A customer pays for one order and gets two receipts. Finance finds two ledger rows for the same payment. Nobody changed the code this week. The payment provider sent the same webhook twice, because the first delivery took too long to answer and the provider retried, and the handler did all its work both times.

That bug is one of the best backend interview exercises we know: small enough for 45 minutes, and hard to fake, because the box replays the duplicates and the database counts the rows. This post is for backend hiring managers: the setup, how we seed it, what strong and weak candidates do in the sessions we review, a time plan and a rubric.

Why duplicates are not an edge case

Webhook senders retry on purpose. Stripe's webhook documentation says plainly that "webhook endpoints might occasionally receive the same event more than once", that Stripe "doesn't guarantee the delivery of events in the order that they're generated", and that in live mode it retries failed deliveries "for up to three days with an exponential back off". It also tells you to return a 2xx quickly, before any slow logic, and lists a timed-out request as a failed delivery (Stripe: Receive Stripe events in your webhook endpoint).

A handler that is correct only when each event arrives once, in order, and quickly, is not correct.

The exercise setup

The service. orders-svc exposes POST /webhooks/payments. On payment.succeeded it verifies the signature, marks the order paid, inserts a row into ledger, and calls a mail stub to send a receipt. On payment.refunded it marks the order refunded and inserts a negative ledger row. Each event carries an id (unique per event), a type, an order_id and a created timestamp.

The seeded bugs. Four, layered so a fix for one exposes the next:

  1. Check-then-insert. The handler does SELECT on a processed_events table, and if the event id is missing, does the work and inserts the id afterwards. There is no unique constraint. Two deliveries that arrive together both see "not processed".
  2. Slow synchronous work. The receipt is sent inside the request, and the mail stub takes about 12 seconds. The replay script, like a real sender, times out after 10 and retries.
  3. Last write wins. A payment.refunded that arrives before a late retry of payment.succeeded is overwritten, and the order goes back to paid.
  4. Wrong status codes. A unique violation (once a candidate adds the constraint) bubbles up as a 500, so the sender retries a duplicate it should have been told was fine. A bad signature also returns 500 instead of 400.

The replay script. tools/replay.py is the provider. It signs events with the box's test secret and has three modes, which the brief lists:

  • --dup sends the same payment.succeeded twice at the same moment.
  • --out-of-order sends payment.refunded first, then the original payment.succeeded.
  • --slow uses a 10 second timeout and retries with backoff, the way a sender does when your handler is slow.

make check runs all three against a fresh database and prints what matters to the business: ledger rows per order, receipts sent, final order status, and the status code returned for each delivery. This is also what the Run tests button on the challenge runs, so the candidate can check their own work as often as they like, and the grader sees the same result.

What strong candidates do

In the sessions we review, the strong ones run make check before touching code. They read the output, see two ledger rows and two receipts, and open the handler with a question already formed.

Then, usually in this order:

  • They make the database the referee. A unique constraint on processed_events.event_id, and the insert of the event id moved into the same transaction as the ledger insert and the status change. INSERT ... ON CONFLICT DO NOTHING and a check of the affected row count is the usual shape. They explain why the original SELECT could never work under concurrency, without being asked.
  • They get the slow work out of the request. The handler writes the business change and an outbox row in one transaction, returns 200, and a worker sends the receipt from the outbox. Some candidates go only as far as a background job; the strong ones notice that a job queued outside the transaction can be lost or sent twice, and say so even if there is no time to build the full outbox.
  • They decide what order means. Rather than trusting arrival order, they make the state machine refuse to move backwards: once an order is refunded, a late payment.succeeded is recorded and ignored. Stripe's page warns against using created to decide order, since distinct events can share a timestamp, and the best candidates reason from the order's state instead of the timestamp.
  • They return codes the sender understands. A duplicate gets a 2xx, because the event was handled. A bad signature gets a 400. A real failure, such as the database being down, gets a 5xx, so the sender retries. They can say in one sentence why each one is what it is.

They also tell you what they did not finish.

What weak candidates do

  • An in-memory set of seen ids. It passes --dup on one process and fails the moment there are two, or a restart. Ask what happens with two replicas; the answer tells you a lot.
  • A lock around the whole handler. It makes the duplicates go away and makes the timeout worse, so --slow now produces more retries.
  • Catching the unique violation and returning 500. The data is right and the sender retries forever. Candidates who stop at "the rows look right" never look at the status codes column.
  • Sorting by created. Works in the sandbox, breaks when two events share a second.
  • Letting the AI write it and not running the replay. The in-session AI panel is on, in Chat, Plan and Agent modes, with a budget ring for the allowance. A model will happily write a plausible idempotency check. Weak candidates accept it; strong ones ask for safety under concurrent duplicates and prove it with the replay. Both show up in the recording beside the terminal.

Time plan for a 45 minute session

Minutes What happens
0 to 5 Read the brief, run make check, describe what is wrong
5 to 20 Fix the duplicate: constraint, atomic insert, transaction boundary
20 to 30 Fix the timeout: move the receipt out of the request, ideally via an outbox
30 to 38 Fix ordering and status codes
38 to 45 Run the full replay, explain what is left and what they would do next

Not finishing all four is fine. The order they choose, and what they say about what they skipped, is signal.

Scoring rubric

Score each row 1 to 4, and write one line of evidence pointing at a moment in the recording.

Area 1 2 3 4
Diagnosis Edited before running anything Found the duplicate only Found duplicate and timeout Named all four before fixing
Idempotency In-memory set or none Check then insert Unique constraint, separate from the work Constraint and work in one transaction
Slow work Left in the request Background job, no thought on loss Background job, risks stated Outbox written in the same transaction
Ordering Arrival order Sorted by timestamp State machine, partly State never moves backwards, explained
Status codes 500 on duplicates Mixed Right for duplicates Right for duplicate, bad signature and real failure, with reasons
Verification Never ran the replay Ran it once at the end Ran it after each fix Ran it after each fix and checked codes, not just rows

A hire for a backend role that touches payments usually scores 3 or more on idempotency and verification. A 1 on idempotency is a no for that role, whatever else went well, because it is the bug that charges customers twice. For more on building rubrics like this, see how to write a technical interview rubric.

Running it on EasyEnv

Each candidate gets their own box in the same broken state, torn down after the interview. The session is recorded, so a second interviewer can watch the moment the candidate read the make check output instead of relying on notes. AI grading gives a first pass on the end state, and the interviewer fills the scorecard, confirming or overruling it. Each interviewer's vote stays hidden until they have cast their own, so the reviewer impressed by a clever lock and the one who noticed the timeout got worse score independently.

What it does not replace: ten minutes of conversation about your queue, your provider and your on-call.

Related: our distributed systems assessment uses a retry that lands twice as one failure mode, and how to assess debugging skill covers the diagnosis half of the rubric.

The takeaway

Take one webhook your system actually receives, write a script that sends it twice, late and out of order, and give a candidate 45 minutes to make the database tell the truth. You will learn more about how they build backends than a whiteboard round can show you.

Which webhook in your stack would fail this replay today?

For hiring teams

Hire engineers with real-world interviews

Run live coding sessions and take-home challenges in real production environments. Watch sessions back, score consistently, and hire with confidence.

Continue reading

More posts you might like