Workflow incident response

Use this runbook when a Trigger stops, runs fail or stall, a provider Action produces an incorrect side effect, or delivery volume differs from the system of record. The priority order is: protect people and data, stop unsafe new side effects, preserve identifiers, determine scope, repair the cause, and recover with one controlled event.

Assign incident ownership

Before production use, name:

• an incident commander who coordinates decisions;
• an automation owner who understands the Workflow graph;
• a provider owner who can inspect credentials, scopes, logs, quotas, and resulting objects;
• a downstream owner who can assess duplicate or missing side effects; and
• a communication owner for affected users or teams.

Severity and paging thresholds are team policy. Base them on business impact, data exposure, irreversibility, duration, and affected volume rather than the presence of one technical error.

First ten minutes

1. Confirm the FastHook team and environment. Do not investigate production identifiers in a test team.
2. Record the detection time, suspected start time, Workflow ID/name, Source ID, provider, and current owner.
3. Search Workflow Activity for the narrowest useful window and filter by failed status or Source.
4. Determine whether events are absent, runs are absent, runs are queued/running, a Step fails, or FastHook succeeds while the provider outcome is wrong.
5. If new side effects are unsafe, pause the Workflow. Disable the Source only when all ingestion and related Connections must stop.
6. Do not replay, manually repeat, reconnect, rotate, or edit the graph until the duplicate-side-effect risk is understood.

Pausing prevents new Workflow execution but should not be treated as cancellation of already queued, running, retrying, or delayed work. Inspect those runs separately.

Triage by symptom

| Symptom | Inspect first | Likely next action | | --- | --- | --- | | Provider events exist but no FastHook activity | Source status, connected account, Instant channel state, provider webhook/subscription | Repair or retry the channel, or use supported polling fallback. | | Requests exist but no Workflow run | Workflow status, selected Source, graph version, request routing | Activate the intended Workflow or correct Source selection after preserving evidence. | | Run remains queued/running | Attempt count, active Delay, Step timestamps, lease/retry timing | Wait for intended delay/retry or escalate a stale run with its identifiers. | | Action returns 401/403 | Account lifecycle, scopes, provider role, object access | Reconnect or restore provider-side permission. | | Action returns 429/5xx | Retryability, Retry-After, provider status, quotas | Reduce load and allow bounded retries; do not reconnect credentials reflexively. | | Run fails after retries | First failed Step, normalized error_code, response status, error route | Fix the cause, assess idempotency, then use a controlled new event or approved recovery. | | FastHook succeeds but outcome is missing | Provider response/output, asynchronous provider logs, object identifier | Reconcile with the provider before repeating the Action. | | Duplicate side effect | Request/run IDs, retries, provider idempotency key, manual actions | Stop repeats and reconcile existing provider objects before compensation. |

Collect minimum safe evidence

Record identifiers and metadata before copying payloads:

• team, environment, Source ID, Workflow ID, and Workflow version;
• request ID, run ID, first failed Step ID, attempt count, and timestamps;
• normalized error code, HTTP response status, and provider key;
• connected-account ID and lifecycle, but never credential values;
• channel status, alert code, failure count, expiration, and last reconciliation time;
• provider request, event, or resulting object ID; and
• mitigation, decision owner, and time.

Use redacted Audit previews only when needed. Audit preview and sanitized-share redaction are key-name based and cannot find every secret in free text. Prefer an expiring sanitized request share over screenshots or raw payload copies, omit the body when it is unnecessary, and add business-specific redaction keys.

Contain safely

Choose the smallest control that stops additional harm:

• Pause the Workflow to stop new runs while preserving the Source and other consumers.
• Disable the Source when inbound processing and related Connections must stop, understanding that provider events may continue outside FastHook.
• Pause or disable a direct Connection when only one delivery branch is unsafe.
• Switch an eligible Instant Trigger to polling when channel failure risks missing events and slower detection is acceptable.
• Revoke or rotate one exposed credential class when there is evidence of compromise.
• Disable a provider-side integration when FastHook controls are insufficient to stop the external effect.

Record queued, running, delayed, and retrying work before containment. A timeout or failed response does not prove the provider made no change.

Repair and recover

1. Fix the smallest confirmed cause: permission, deleted resource, invalid mapping, provider quota, channel subscription, receiver availability, or graph logic.
2. If the graph changed, save a new forward version and record the previous and new version numbers.
3. If credentials changed, follow the appropriate rotation procedure and verify the connected account or receiver independently.
4. Decide whether missing work should be recreated from the provider, sent as a new controlled event, retried, replayed, or reconciled manually.
5. Use an idempotency key or stable business identifier wherever the provider supports one.
6. Run one recognizable canary through the complete path.
7. Verify FastHook Audit, the final provider state, and the business system of record.
8. Resume traffic gradually and watch the first production runs.

FastHook automatically retries retryable Workflow failures for up to ten queue retries. When delivery is exhausted, the run and pending/running Step records are closed as failed with workflow_delivery_exhausted. Do not add an uncontrolled manual replay loop on top of automatic retries.

Escalation package

When escalating to another team or FastHook support, provide:

• a concise impact statement and UTC time window;
• sanitized team, Source, Workflow, request, run, and Step identifiers;
• status and normalized error code;
• whether the problem is Trigger, execution, delivery, provider, or business-outcome related;
• the last known successful event and first known failure;
• containment already applied; and
• an expiring sanitized share only if payload evidence is necessary.

Never send project keys, signing secrets, OAuth tokens, provider API keys, raw cookies, authorization headers, or unredacted customer payloads.

Close and review

Close the incident only after the cause is fixed, canary and production evidence agree, backlogged work is reconciled, alert delivery works, and temporary credentials or fixtures are removed.

The post-incident review should record:

1. impact and exact timeline;
2. detection source and why it was or was not timely;
3. contributing Trigger, graph, provider, retry, permission, or process conditions;
4. duplicate and missing-event reconciliation;
5. corrective changes with owners and due dates;
6. monitoring or alert changes; and
7. documentation, production-checklist, and change-control updates.

Continue with Workflow monitoring and alerts, Instant Trigger recovery, and Security and governance.