Building resilient webhook integrations

A webhook endpoint sits on a distributed delivery boundary. Duplicates, reordering, delayed events and supplier retries are expected operating conditions rather than exceptional corruption.

Authenticate the original bytes

Verify the supplier signature against the unmodified request body, use constant-time comparison and reject timestamps outside an agreed window. Rotate secrets with an overlap period and record which key verified the request.

Acknowledge after durable receipt

Validate the envelope, persist the event or place it on a durable queue, then return the expected success status quickly. Do not hold the supplier connection open while updating customer records or calling another external service.

Make processing idempotent

Store the provider event identifier and result in the same transactional boundary as the business change. If no identifier exists, construct a documented deduplication key without treating two legitimate similar events as duplicates.

Handle ordering explicitly

Use resource versions, occurrence timestamps or a reconciliation API when later events can arrive first. A queue preserves local processing order but cannot repair the order in which a remote producer delivered events.

Operate replay and reconciliation

Retain payloads according to data policy, expose failed events with reason and attempt count, and provide a controlled replay action. Run scheduled reconciliation for states whose loss would matter even when no webhook failure is visible.