How do I use Inngest Replay or Bulk Cancellation to recover from a bad deploy or broken workflow?

Bad deploys and broken workflows are inevitable when you’re shipping fast—especially with multi-step, event-driven systems. The real question isn’t “how do I avoid failure forever?” but “how quickly can I see what went wrong, stop the bleeding, and safely recover state without ad‑hoc scripts or dangerous one-off jobs?”

Inngest Replay and Bulk Cancellation exist for exactly this: turning “oh no” moments into controlled, auditable recovery flows.

This guide walks through how to use Inngest Replay and Bulk Cancellation to recover from a bad deploy or broken workflow, and how to think about them as core parts of your incident playbook.

Why Replay and Bulk Cancellation exist

In most queue/worker stacks, a bad deploy looks like this:

Some percentage of events start failing a multi-step workflow.
Partial state gets written: some steps ran, some didn’t, retries are inconsistent.
You ship a fix, then:
- Manually dig through logs to find impacted runs.
- Hand-roll scripts to replay “good” events.
- Write one-off SQL or admin jobs to “clean up” bad state.

You’re juggling dead-letter queues, retry semantics, and ad-hoc trace IDs—exactly the kind of infrastructure tax Inngest is designed to remove.

Inngest’s durable execution engine and step-level checkpointing change the recovery story:

Each step.run() is a named, durable unit of work with automatic retries and checkpointing.
Traces give you step-by-step inputs/outputs and failure reasons, in one place.
Replay and Bulk Cancellation let you query, cancel, or replay thousands of runs without building your own admin system.

Put simply:

Replay = “Re-run this workflow (or slice of workflows) using the latest code and configuration.”
Bulk Cancellation = “Stop all current and future progress for a set of runs you no longer trust.”

Used together, they give you a safe, repeatable path to recover from a bad deploy or broken workflow.

Core concepts: what Inngest is actually doing for you

Before we go step‑by‑step, it helps to anchor on the mechanics.

Durable execution and checkpointing

Inngest workflows are defined in code using inngest.createFunction() with Steps like:

import { inngest } from "./client";

export const syncCustomer = inngest.createFunction(
  { id: "sync-customer" },
  { event: "customer.updated" },
  async ({ event, step }) => {
    const customer = await step.run("fetch-customer", async () => {
      return await fetchCustomer(event.data.id);
    });

    await step.run("update-crm", async () => {
      return await updateCrm(customer);
    });

    await step.run("send-webhook", async () => {
      return await sendWebhook(customer);
    });
  }
);

Key behavior:

Each step.run() is a transactional boundary.
On failure, Inngest retries the failing step automatically.
When a step succeeds, Inngest checkpoints and never re-runs it for that workflow run.
On Replay, the workflow resumes from the last successful step, not from the beginning, unless you explicitly reset state.

This is why replaying thousands of runs is safe—you’re not re‑doing already completed work unless you choose to.

Traces: your incident console

For every function run, Inngest Cloud surfaces:

Real-time traces of each step.
Step-level inputs and outputs.
Structured logs tied to the run.
Final status (succeeded, failed, cancelled, etc.).

This is where you identify the blast radius, filter the impacted runs, and choose between Replay and Bulk Cancellation.

Typical incident flow: bad deploy or broken workflow

When a bad deploy hits, I think in four phases:

Detect: You see errors spike in Traces, alerting, or your own app.
Scope: You figure out which workflow(s), tenants, or time ranges are affected.
Fix: You roll forward (new deploy) or roll back code/config.
Recover: You use Replay and/or Bulk Cancellation to clean up and restore state.

The rest of this article focuses on step 4: how to use Inngest’s built‑in controls to recover.

Option 1: Using Inngest Replay to re-run failed or impacted workflows

Replay is for cases where the workflow is conceptually correct, but a bad deploy, dependency issue, or transient failure led to bad runs. Once your fix is live, you want those runs to complete successfully.

When to prefer Replay

Use Replay when:

You shipped a bad deploy that caused step failures (e.g., bad API base URL, schema mismatch, auth issue).
External dependencies were down (CRM, LLM provider, payment gateway).
You discovered a recoverable bug and have now patched it.
State is mostly correct, and you want to complete partially executed workflows, not discard them.

Avoid Replay when the workflow logic itself produced irreversible side effects you don’t want repeated (e.g., double‑charging customers) unless your code is idempotent and safe to replay.

Step 1: Identify impacted runs in Traces

Open Inngest Cloud.
Go to Traces (or the Functions view, depending on your UI entry point).
Filter by:
- Function: e.g., sync-customer.
- Status: failed, or cancelled if the old code explicitly aborted.
- Time range: window of the bad deploy.
- Additional filters: like event fields (tenant ID, region, feature flag) if you need to isolate specific customers or cohorts.

You’re building a query that expresses: “Show me the runs I want to fix.”

Step 2: Confirm you’ve deployed the fix

Replay uses your current function code and configuration.

If you’re rolling forward, deploy the fixed version of your function.
If you need to roll back, restore a known-good version and deploy that.

Confirm in dev first:

npx --ignore-scripts=false inngest-cli dev

Run a local replay (or a test event) to verify you’ve actually fixed the failure path before you perform a large Replay in production.

Step 3: Replay runs from the UI

In the Inngest dashboard:

From your filtered list of runs, select:
- A single run if you’re testing.
- A set of runs if you’re ready for bulk recovery.
Choose Replay.
Confirm the action—this kicks off new runs for each selected execution.

Under the hood:

Inngest creates new runs tied to the same originating events.
Execution uses the latest code and configuration.
Checkpointing ensures already-completed steps aren’t re‑run unless state has changed and you’ve coded accordingly.

Step 4: Monitor replayed runs

Use Traces to confirm recovery:

Watch the new runs as they progress step by step.
Confirm critical steps (e.g., update-crm, send-webhook) now succeed.
Validate side effects in your own systems (CRM, billing, downstream services).

If something still fails, iterate on the fix and replay again using the same pattern. Because Inngest is durable and step‑aware, replaying isn’t a risky “start over and hope” operation—it’s a controlled resume.

Option 2: Using Bulk Cancellation to stop bad or unsafe workflows

Bulk Cancellation is for stopping workflows you no longer trust, not fixing them. You’re treating those runs as “do not continue”—maybe because they’re causing harmful side effects or because you plan to handle recovery manually.

When to prefer Bulk Cancellation

Use Bulk Cancellation when:

The workflow logic is fundamentally wrong for a cohort (“wrong pricing logic,” “wrong feature flag branch,” “wrong tenant mapping”).
Continuing current runs risks more damage (e.g., sending incorrect emails, writing invalid data, overloading a downstream system).
You want to freeze the current state, investigate, and then make an explicit recovery plan.
You’re deprecating a workflow and want to cancel all in‑flight runs.

You can always follow Bulk Cancellation with a carefully scoped Replay later (for some or all runs) once you’ve fixed the underlying issue.

Step 1: Filter to the runs you need to stop

In Traces or the Functions view:

Filter by Function, Status, and Time range.
Narrow by:
- Tenant/Customer: via event payload fields.
- Region or environment: e.g., only prod-eu.
- Feature flags or version markers in your events.

This query should represent: “These runs should never continue.”

Step 2: Bulk-cancel from the UI

From the filtered results:

Select all relevant runs (or use the “select all” feature if available).
Choose Bulk Cancellation.
Confirm the action.

What happens next:

Inngest marks these runs as cancelled and will not execute further steps.
Downstream retries for these runs are halted.
You keep a record in Traces for audit and postmortem analysis.

Cancellation is durable—you’re explicitly telling the engine to stop, so you can treat these runs as completed in a “do not proceed” state.

Step 3: Decide on a follow-up path

After cancellation, you have options:

Replay later with fixed code: Once the workflow is safe, selectively Replay cancelled runs you want to restore.
Create a one-off migration workflow: Use a dedicated Inngest function to repair state based on the cancelled runs.
Do nothing: In cases where the cancelled runs truly represent work you don’t want to complete.

The key is that Bulk Cancellation buys you time and safety—no more bad work progresses while you reason about recovery.

Combining Replay and Bulk Cancellation in real incidents

Most real incidents mix both tools. A few patterns I’ve seen in production:

Pattern 1: “Bad step logic; external state is mostly intact”

Symptoms: A step like update-crm fails for a subset of runs due to a schema change.
Action:
1. Deploy a fix for update-crm.
2. Filter all sync-customer runs that failed on that step during the bad window.
3. Replay those runs.
Outcome: Completed workflows resume from the failed step, update the CRM, and move on. No need for cancellations.

Pattern 2: “Dangerous side effects; stop everything first”

Symptoms: A deploy accidentally sends the wrong email template or applies incorrect discounts.
Action:
1. Filter runs in the email/discount workflow for the impacted window and tenant cohort.
2. Use Bulk Cancellation immediately to stop further execution.
3. Fix the code and templates.
4. Decide which runs to Replay (if any) vs. handle manually.
Outcome: Damage stops quickly; Replay is used carefully for selected runs, not as a blanket fix.

Pattern 3: “Multi-tenant noisy neighbor; overloaded dependency”

Symptoms: One tenant’s surge saturates an external API, causing widespread failures.
Action:
1. Use Flow Control (concurrency keys, throttling) to protect the multi-tenant workload going forward.
2. For runs that failed due to rate limits, Replay once flow control is in place.
3. For obviously invalid runs (e.g., bad tenant config), Bulk Cancel.
Outcome: Future runs are protected by Flow Control; past runs are selectively recovered via Replay.

Replay and Bulk Cancellation are incident tools, but they’re most powerful when paired with Inngest’s multi-tenant Flow Control—so you don’t just recover from a bad deploy, you harden the system against the next one.

Best practices for safe recovery

A few habits that make Replay and Bulk Cancellation low‑stress instead of scary buttons:

1. Design for idempotency at the step level

When you write steps, assume they might run more than once in edge cases:

await step.run("charge-customer", async () => {
  if (await alreadyCharged(invoiceId)) return;
  return await stripe.charges.create({ /* ... */ });
});

Mechanism → outcome:

Checking alreadyCharged makes replaying safe.
You can confidently Replay runs without worrying about double charges.

2. Use clear, descriptive step names

Good step names pay off in incidents:

validate-input
fetch-customer-from-crm
apply-discount-rules
send-order-confirmation-email

When you’re in Traces, you can see exactly where runs failed, and you can target replays more precisely.

3. Tag events with recovery-friendly metadata

Include fields in your events that help you slice runs later:

{
  "name": "order.created",
  "data": {
    "orderId": "ord_123",
    "tenantId": "tenant_abc",
    "region": "eu",
    "featureFlags": ["new_pricing_v2"]
  }
}

Now you can filter runs by tenantId, region, or featureFlags when performing Replay or Bulk Cancellation.

4. Treat “Replay in staging” as part of your deploy process

Before doing a large Replay in production:

Use the same filters in staging.
Replay a representative batch.
Confirm the workflow completes and external systems are in the state you expect.

This keeps your production replays focused and safe.

5. Keep observability and governance in mind

In regulated or enterprise environments, Replay and Bulk Cancellation are also governance surfaces:

Traces provide a durable record of what was cancelled or replayed.
You can correlate recovery actions with your own audit logs and incident reports.
For teams using Inngest with SOC 2 Type II, SSO/SAML, and HIPAA BAA, these controls fit into your existing compliance posture.

Putting it all together

Using Inngest Replay and Bulk Cancellation to recover from a bad deploy or broken workflow looks like this in practice:

Detect and scope the issue using Traces and your own alerting.
Decide intent:
- Want runs to complete with new logic? → Replay.
- Want runs to stop and never continue? → Bulk Cancellation.
Filter precisely by function, time range, tenant, and flags to isolate the blast radius.
Apply the control (Replay or Bulk Cancellation) from the Inngest UI.
Monitor recovery via step-level Traces and your external systems.
Harden for next time using Flow Control, idempotent steps, and better metadata.

You don’t need to re‑build workers, one-off scripts, or admin tools just to recover from deploys gone wrong. With Inngest, durability is expressed in code, and recovery is a first‑class action—not a late-night log‑grepping exercise.

Next Step

Get Started

Answers you can trust, from Codeables