Transactional email & event reliability

A webhook tells you what happened, not what is true now.

The same event may knock twice, and yesterday’s fact may arrive after today’s. Reliable systems preserve the evidence before deciding what it means now.

By Moosewave

Published August 9, 2026 · 10 min read

Several deliveries can carry one event. Record the evidence once, then update each kind of state deliberately.

The short version

Store the fact before you act on it.

An email webhook is an HTTP notification about an event such as delivery, delay, complaint, unsubscribe, open, or click. It is not a live copy of current message or recipient state.

01Verify the sender against the original request bytes before parsing the payload or changing anything.
02Record a stable source and event ID durably, then return success without waiting for every downstream workflow.
03Expect valid retries and surprising arrival order; make both the handler and each business effect idempotent.
04Build transport, safety, and engagement views from the event log, then reconcile important records with the source.

The outward half of this contract is explained in A retry is not permission to send twice.

One conversation contains four different things

Your application requests an order receipt. The email API accepts it. Later, an email service observes delivery and calls your webhook endpoint. If the response disappears, it calls again. The two HTTP requests look like two things happening. They may be two attempts to report one thing.

This is the mirror image of uncertainty on the send path. A stable idempotency key stops repeated API attempts from becoming repeated email. A stable event identity stops repeated webhook attempts from becoming repeated workflow changes. Both are needed because a network can lose an acknowledgement without losing the work it acknowledges.

Occurrences, events, delivery attempts, and projections
Object	Meaning	Example
Occurrence	A fact observed in a system	A receiving server accepted the message
Event	A record describing that fact	delivery.completed / evt_71
Delivery attempt	One transport attempt carrying it	POST number 1, 2, or 3
Projection	The current view derived from facts	Recipient transport: delivered

CloudEvents makes the distinction explicit: an event is a record of an occurrence, and one occurrence can yield more than one event. It defines source plus id as the event identity. Neither an HTTP request ID nor a hash of the whole payload is a universal substitute for the provider’s documented identity.

The webhook is a delivery mechanism for evidence. It is not the database where truth lives.

Make the receiver boring and fast

A webhook endpoint should do a small amount of important work. It should prove who sent the request, preserve what was sent, and make later processing possible. Rendering a report, updating six systems, or sending another message can happen after the provider connection has closed.

01
Preserve the original bytes
Signature schemes commonly cover the request body exactly as sent. Parsing and re-serialising it first can change the bytes required for verification.
02
Verify source and recency
Use the provider's signature header, endpoint secret or public key, and replay-window rules. Reject invalid requests before any side effect.
03
Insert one immutable event
Store workspace, source, event ID, type, schema version, occurrence time, message and recipient identity under a unique constraint.
04
Enqueue, then acknowledge
Commit a processing intent with the event, or make the durable queue the capture boundary, then return 2xx. A valid duplicate can receive the same success.

webhook receiver / pseudocodeverify · persist + schedule · acknowledge

raw = request.body_bytes
verify(raw, signature, timestamp)
event = decode(raw)

transaction:
  created = events.insert_once(
    key = [workspace, event.source, event.id],
    envelope = event
  )
  if created: outbox.insert_once(event.id)

return 204

A relay publishes committed outbox rows to the worker queue. This closes the crash gap between storing the event and scheduling its processing; a durable queue can serve as the same boundary when its write is the first committed record.

Stripe documents automatic retries, duplicate events, unordered delivery, signature verification, asynchronous queues, and prompt successful responses. GitHub similarly recommends a quick 2xx and a processing queue. Exact timeout and signature formats differ, but the boundary remains: authenticate and capture before acknowledging; do expensive work later.

Keep endpoint secrets scoped and rotatable, map each event to the correct workspace, and retain an audit trail. Those controls belong beside the isolation boundaries on the Moosewave security page.

Deduplicate twice

The first deduplication answers a transport question: have we recorded this event before? A unique constraint on source and event ID makes that decision atomic. Two workers can race and still leave one record.

The second answers a product question: has this effect already happened? Recording evt_71 once does not help if a worker suppresses a recipient, publishes a notification, and crashes before marking the job complete. A retry can run the business action again.

Event key: workspace + provider source + provider event ID
Projection key: message + recipient + projection version
Effect key: business action + stable source fact
Replay result: the existing outcome, not a second effect

Where possible, commit the state change and effect record in one transaction. When the effect lives elsewhere, use a durable inbox or outbox and give the next operation its own idempotency key. AWS advises idempotent event consumers because duplicate processing can occur in managed systems.

Do not discard every similar payload as a duplicate. Stripe notes that separate event objects can sometimes describe the same object change. Preserve both records, then make the domain effect safe using documented object, event-type, recipient, and revision fields.

Arrival order is not event order

Webhooks travel on different schedules. An endpoint outage can leave one event in retry while a newer event succeeds on its first attempt. The HTTP arrival time describes your connection, not the chronology of the customer’s experience.