Webhook Idempotency and Retries: Keys, Windows, Side Effects
Direct answer. Most senders retry on timeouts, 5xx and network errors, and the rest expect you to redeliver. Treat every delivery as at-least-once: key each event (provider + event id, or Idempotency-Key), persist processing state before side effects, and keep the dedup store at least as long as the sender’s maximum retry span. Completed keys get a plain 2xx.
1.0 Senders retry; receivers must assume at-least-once
A webhook that timed out on your side may already have been applied. The sender does not know that, so it retries. A 500 or a plain network failure triggers the same retry. Designing as if each event arrives exactly once is how you get duplicate emails, double stock movements or two ERP documents for one payment.
Providers differ on how long they keep trying. Stripe, for example, retries live webhook deliveries for up to three days with exponential backoff. GitHub does not redeliver failed deliveries automatically; you can redeliver anything from the past three days in the UI or API. Read the provider’s retry and redelivery notes and size your dedup retention against the longest window you care about, not against a guess that “we answer quickly so they will not retry”.
2.0 Choose a stable key
| Source | Typical key | Notes |
|---|---|---|
| Provider event id | evt_…, delivery GUID | Prefer the provider’s own event id when it is unique per logical event |
| Composite | (provider, event_id) | Use when one system accepts webhooks from several providers |
| Header | Idempotency-Key, X-GitHub-Delivery | Idempotency-Key is for your own outbound APIs. GitHub keeps the same X-GitHub-Delivery GUID on redelivery, so it identifies the event, not the attempt |
| Business object | (event_type, object_id) | Needed when the provider may emit two Event objects for the same underlying change |
Store the key with a unique constraint (or an atomic conditional write). Concurrency is the hard case: two workers can both see “missing” unless the insert itself is the gate.
3.0 Track a state, not a done flag
| State | Meaning | On a new delivery of the same key |
|---|---|---|
| processing | Accepted, work in flight (optional lease / owner) | Wait, return conflict, or attach to the in-flight attempt — do not start a second side-effect path |
| completed | Persisted and side effects done | Return 2xx immediately; no side effects |
| failed_retryable | Soft failure; safe to try again | Retry under backoff; still under the same key |
| failed_terminal | Will not succeed without a person | Alert; do not auto-loop forever |
Mark processing in the same durable write that first accepts the key, before any email, charge or ERP call. Mark completed only after those side effects have a durable record of their own (or after the outbox row that will drive them is committed).
4.0 Retry window versus dedup retention
Two durations have to line up:
- Sender retry window — how long the provider (or your redelivery script) may send the same event again.
- Receiver dedup retention — how long you keep keys and outcomes so a late duplicate is recognised.
Retention must be ≥ the retry window you accept. If keys expire earlier, a late retry looks new. Manual redelivery usually reaches further back: Stripe retries automatically for up to three days, but the Dashboard can resend an event for 15 days and the CLI for 30. If operations still re-push from the dashboard after your retention ends, either extend retention or require a fresh key and an explicit reconcile path.
5.0 Persist before you acknowledge
The awkward case is: your handler finished the work, then the connection dropped before the sender saw 2xx. The sender retries. If you only remembered success in memory, you will run the side effects twice.
The order we write things in:
- Write the event (or an outbox row) and the processing / completed state first.
- Return 2xx only after that write commits.
- Drive non-idempotent side effects from the durable record, after the dedup gate.
- If side effects are slow, acknowledge after durable accept and finish them asynchronously — still under the same key and state machine.
6.0 Side effects sit behind the gate
Anything that is hard to undo — send email, charge a card, push an ERP document, open a ticket — runs only after the key is claimed. Replays of a completed key skip that path and return success. If a side effect itself is not idempotent at the far end, give it its own business key (order number, invoice id) so a second push does not create a second document.
7.0 Ordering under at-least-once delivery
Providers often do not guarantee order. An updated event can land before created, or a retry of an older payload can follow a newer one. Idempotency alone does not fix that. Keep a version, sequence or updated_at per business object and ignore stale writes, or fetch the current resource from the provider before changing local state.
8.0 What to log and alert on
- Index the dedup store by key so “did we already handle
evt_…?” is one query. - Log each attempt: key, outcome (duplicate / processing / completed / failed), latency, attempt count.
- Payment or personal data stays out of shared logs; we log a digest and our internal id.
- Alert on keys stuck in processing past the lease, and on sudden spikes of duplicates (often a sender retry storm after an outage).
If you are building webhook receivers for payments, ERP or device clouds, see Web, Cloud and Device Management, or discuss a project with us.
References
- Stripe, Receive Stripe events in your webhook endpoint: https://docs.stripe.com/webhooks
- Stripe, Idempotent requests: https://docs.stripe.com/api/idempotent_requests
- GitHub, Best practices for using webhooks: https://docs.github.com/en/webhooks/using-webhooks/best-practices-for-using-webhooks
- GitHub, Redelivering webhooks: https://docs.github.com/en/webhooks/testing-and-troubleshooting-webhooks/redelivering-webhooks
- IETF HTTPAPI WG, The Idempotency-Key HTTP Header Field, draft-ietf-httpapi-idempotency-key-header-07 (expired Internet-Draft): https://datatracker.ietf.org/doc/draft-ietf-httpapi-idempotency-key-header/
Frequently asked questions
Our provider says they deliver each event once. Do we still need a dedup store?
At-least-once is the common contract: timeouts, 5xx responses and network blips all trigger retries. A stable key plus a retention window that covers the provider's maximum retry span is how you avoid double side effects when the same payload arrives again.
What should we return when a duplicate arrives while the first attempt is still processing?
Return a conflict or a short wait signal according to what the sender documents, and never start a second side-effect path for the same key. We hold the row in processing with a lease (owner and expiry), and mark it completed only after persistence and side effects finish.
How long should we keep processed keys?
At least as long as the sender's automatic retry window, and preferably through any manual redelivery period they publish. If a key expires and the same event lands again, the receiver will treat it as new and may repeat non-idempotent work.
We already returned 200, but the email or ERP push never ran. How do we recover?
Persist the event and its processing state before returning 2xx, then drive side effects from that durable record or an outbox. Return success before the write lands and a crash right after leaves nothing to recover from, while the sender already counts the event as delivered.
Events sometimes arrive out of order. Does the idempotency key fix that?
The key only prevents double application of the same event. Ordering needs separate rules: ignore stale versions with a sequence or updated-at check, or fetch the current object from the provider's API before mutating local state.