Webhook Idempotency And Deduplication Patterns

What FastHook Deduplication Does

A deduplicate rule runs on a connection before FastHook sends the event to that connection's destination. The first matching payload in the window is allowed through. Later payloads with the same dedup key are treated as duplicates for that connection.

• The first event creates the normal delivery path and can produce destination attempts.
• A duplicate creates an ignored event with reason deduplicate and a link to the previous event when FastHook still has it.
• A duplicate does not create a destination delivery attempt, because FastHook does not call the destination for that connection.
• Deduplication is scoped per connection, so the same source event can be deduplicated for one destination and still be delivered to another.
• Deduplication is skipped for retry traffic, so retrying a failed event can still resend it to the destination.

Rule Shape

Add a deduplicate rule to the connection that should suppress duplicate deliveries. The window is in milliseconds. FastHook currently accepts windows from 1000 ms to 3600000 ms.

The dashboard Deduplicate block stores the same backend rule shape: type deduplicate, window, and either include_fields, exclude_fields, or neither for whole event data.

• type must be deduplicate.
• window is the first-seen duplicate window in milliseconds.
• include_fields builds the dedup key from only those request-envelope paths.
• exclude_fields builds the dedup key from normalized request data after removing volatile paths.
• If neither include_fields nor exclude_fields is provided, FastHook hashes the whole normalized body.

{
  "type": "deduplicate",
  "window": 300000,
  "include_fields": ["body.id"]
}

Choose The Dedup Key

Prefer an id that the provider promises is stable for the business event. A provider delivery id is often not the same thing: delivery ids can change on each retry, while event ids should stay stable.

When the provider has no single event id, combine fields that identify the business action, such as object id plus event type. Keep the key narrow enough to ignore timestamps and metadata, but specific enough that different events do not collapse into one duplicate.

• Best default: include the provider event id, for example body.id or body.event_id.
• For Stripe-style envelopes, deduplicate by the provider event id rather than by the nested object id alone.
• For GitHub-style payloads, use the stable event identity available in the body when present, and keep the receiver idempotent because some GitHub metadata can change between deliveries.
• For internal webhooks, create a durable idempotency key at the producer and include it in the body.
• Do not include timestamps, retry counters, request ids, or generated delivery ids unless they are part of the business identity.

Field Paths

Deduplication field paths are request-envelope paths. Use body.* for parsed body fields, headers.* for headers, query.* for query params, and path or method when those values are part of the duplicate identity.

Array wildcard paths are supported. The backend accepts body.items[*].id and keeps it in the saved include_fields array.

• body.id reads a top-level id from the JSON body.
• body.data.object.id reads a nested object id.
• body.items[0].id reads the first item in an array.
• body.items[*].id reads matching values across every item in an array.
• headers.content-type and headers.stripe-signature are valid request-envelope paths when a header is part of the key.
• If an include_fields path is missing, its value becomes an empty match set, so choose fields that are always present for the events using this connection.

{
  "type": "deduplicate",
  "window": 300000,
  "include_fields": [
    "body.id",
    "body.type",
    "body.data.object.id",
    "body.items[*].id"
  ]
}

Include Fields Example

Use include_fields when the provider exposes a stable event id. This is the safest pattern because small metadata changes in the rest of the payload will not change the dedup key.

• The first payload with body.id equal to evt_123 is delivered.
• Another payload with body.id equal to evt_123 inside the 5 minute window is ignored for this connection.
• A payload with body.id equal to evt_456 is a different event and is delivered.
• A manual retry of the first FastHook event is still allowed because retries bypass deduplication.

{
  "type": "deduplicate",
  "window": 300000,
  "include_fields": ["body.id"]
}

Exclude Fields Example

Use exclude_fields when the full body is mostly stable but the provider adds volatile values on every send. FastHook hashes the normalized body after removing the excluded paths.

• This pattern is useful for providers that resend the same object with a new delivery id.
• Exclude only fields that are not part of the business meaning of the event.
• If the provider can add new volatile fields later, include_fields is usually more predictable than exclude_fields.

{
  "type": "deduplicate",
  "window": 300000,
  "exclude_fields": [
    "body.sent_at",
    "body.delivery_id",
    "body.retry_count"
  ]
}

Pick The Window

The window starts when FastHook accepts the first event for the dedup key. A duplicate inside the window does not extend the window; expiration is based on the first accepted event.

• Use 30000 to 300000 ms for short provider retries and accidental double sends.
• Use 600000 to 3600000 ms when the provider may replay the same event over a longer recovery window.
• Avoid very long windows for high-cardinality or mutable events unless the provider event id is truly stable.
• If duplicates can arrive after the FastHook window, the receiver still needs its own idempotency storage.

Connection-Scoped Fan Out

Deduplication belongs to a connection, not to a source. This is important for fan-out: each destination can choose its own duplicate policy.

• Billing may suppress provider retries so customers are charged once.
• Analytics or audit destinations may choose to receive every delivery or use a different window.
• The same body.id can be a duplicate on one connection and a first delivery on another connection.

Source: stripe-production
  -> Connection: billing-api     deduplicate by body.id
  -> Connection: warehouse-sync  no deduplication
  -> Connection: audit-log       deduplicate by body.id with a longer window

Retries And Manual Replay

FastHook does not apply deduplication to retry events. This keeps recovery explicit: if a destination failed and you retry the event, FastHook should be able to send the original event again.

• Automatic destination retries are about recovering a failed delivery attempt, not suppressing a new inbound duplicate.
• Manual retry and bulk retry can resend an event even if its original payload would match a dedup key.
• Your destination should store provider event ids or business idempotency keys before applying side effects.
• Use FastHook deduplication to reduce obvious duplicate traffic; use receiver idempotency to protect money movement, inventory updates, account changes, and other irreversible work.

How Duplicates Look In FastHook

When a connection suppresses a duplicate, FastHook records it as an ignored event instead of creating a destination attempt. That gives operators a traceable reason without making the downstream service handle the duplicate.

• The ignored reason is deduplicate.
• duplicate_of_event_id points at the first event when FastHook can associate it.
• There is no destination response because the destination was not called.
• The original request can still create events for other connections that are not deduplicating the same key.

Checklist

• Use include_fields with a provider event id whenever possible.
• Use exclude_fields only when you understand every volatile field that should be ignored.
• Keep the dedup window as short as your provider retry behavior allows.
• Do not deduplicate on a field that is optional or often missing.
• Remember that deduplication is per connection and retry traffic bypasses it.
• Keep destination handlers idempotent even when FastHook deduplication is enabled.

Next

• Fan-out webhooks
• Connections
• Retries
• Verify webhooks