Pausing autonomous agents for a human, durably

Context

I wrote the spec for and led Human Review in Clearskies Workflows: actions that touch Salesforce, Slack or email can pause until a named reviewer approves them. It came to about 70 PRs across the frontend and backend in five weeks.

The need is easy to state. Letting an agent write to a company’s system of record, or message its customers, with nobody looking is a lot to ask of the people responsible for that data. They want to say “do all of this automatically, but show me these specific actions before they happen.” Without that checkpoint, teams only automate low-risk work.

This page is about the engineering patterns rather than the product: what it takes to pause a durable workflow for a human, and the guards I’d put around it on any engine like Temporal.

The constraints

Pausing is easy to say. On a durable workflow engine, it comes with some hard rules:

• The pause can last as long as the reviewer takes. Hours, a weekend, whatever. The run can’t hold a worker or an open activity while it waits.
• Deploys happen while runs are paused. Temporal rebuilds a workflow’s state by replaying its history against the current code. Change the code a paused run will replay through, and its history no longer matches.
• The reviewer has to approve exactly what will happen. If the action is recomputed after approval, against data that may have changed, the reviewer approved something else.
• Nothing gets applied twice, and nothing unreviewed gets applied at all. A duplicate Slack post is embarrassing. A duplicate or unreviewed CRM write is a support ticket.
• The payload is sensitive. A pending email or record update holds customer data, and it’s now sitting in a review queue.

Options

Where to wait. You can tear the run down at the review point and re-enqueue it after the decision, or keep it as one live execution that blocks on a signal. Re-enqueueing means rebuilding the run’s position from storage. Blocking in workflow code keeps all of that for free, costs nothing while parked, and survives worker restarts. A wait inside an activity, by contrast, is bounded by the activity’s timeout. So the pause belongs in workflow code, ideally at the one place every node passes through on its way to running.

What gets approved. Either re-run the action after approval, or compute it once, show the reviewer that snapshot, and apply the snapshot. Re-running is simpler but breaks the “approve exactly what happens” rule. So every action splits into two halves: resolve (work out exactly what would be written or sent) and apply (do it). Approval applies the stored snapshot.

What “reject” means. It’s tempting to reuse whatever early-exit path the engine already has. But a rejected run must never be reported as completed, so rejection gets its own terminal status. Rejecting skips the action and stops the run.

Where the decision lands first. When a reviewer clicks Approve, the API can record the decision and then tell the workflow, or tell the workflow first and record the decision only once that succeeds. Signal first is better: if the run is already gone, the API finds out immediately and marks the review abandoned, instead of leaving an “approved” that will never happen. There’s still a window where a decision is recorded but the signal is lost, so you build a reconciler for that window rather than pretending it doesn’t exist.

What not to support yet. Review inside a parallel loop is subtle: parked iterations share a signal channel, and “reject stops the run” doesn’t hold when siblings keep running. Rejecting that configuration at validation time is cheaper than shipping something subtly wrong. Sequential loops are fine: one review per iteration.

How to ship into live runs. Wait for a quiet moment with no runs in flight, or version the workflow code. There’s never a reliably quiet moment, so: versioning.

The call

The shape is a small state machine inside the run.

In order, when the run reaches an action that needs review:

1. Resolve. The action works out exactly what it would do (the field values, the message, the email), and that becomes the snapshot. Splitting resolve from apply across every action is the unglamorous prerequisite for everything else.
2. Pause. An activity records the pending review and parks the run. It’s idempotent, so a retry doesn’t create a second review item, and it’s capped: past a limit on parked runs, it fails closed. A run that can’t be reviewed doesn’t get to skip review.
3. Wait. The workflow waits on a signal. Waiting costs nothing, and it survives worker restarts and deploys.
4. Decide. The reviewer approves or rejects. The API delivers the decision to the workflow as a signal first, and records it only if that worked.
5. Apply or stop. On approval, the run applies the stored snapshot, not a fresh resolve, and an apply marker makes it at most once even if the decision arrives twice. On rejection, the action is skipped and the run ends as rejected.

The versioning piece looks like this, in spirit:

v = workflow.GetVersion(ctx, "review-pause", DefaultVersion, 1)
if v == DefaultVersion:
    runActionAsBefore(step)        # histories recorded before the deploy
else:
    snapshot = resolve(step)
    if needsReview(step):
        parkAndWaitForDecision(snapshot)
    apply(snapshot)

What breaks, and the guard for each

A rejected run reads as completed. Reusing an existing early-exit path risks exactly that. Give rejection its own terminal status, and test that it reads as rejected everywhere.

Signals go missing. Any time two systems record a fact, they eventually disagree: here, the workflow’s view of the review and the database’s. A reconciler finds runs whose decision was recorded but never delivered, and re-delivers it. Re-delivery is only safe if applying is at most once, so the apply step checks a marker, and a metric flags any attempt to apply twice.

The reconciler cries wolf. A run with several review steps parks again at each one. A naive lost-signal scan sees an earlier, already-honored decision next to a parked run and calls it lost. Only consider decisions that haven’t been applied yet, and pin a re-parking run in tests.

Replay against new code. Without a version gate, the first deploy of the pause logic breaks every run already in flight. With it, old histories take the old path.

Parked runs pile up. A misconfigured reviewer or an ignored inbox means runs accumulate. A cap stops the growth, and the failure mode is closed: the action doesn’t happen.

Retries duplicate things. Activities retry. A pause that isn’t idempotent turns one review into several.

A new status ripples. Anywhere the set of “active” statuses is repeated (constraints, indexes, queries that block duplicate triggers), a paused run has to be added in the same change.

Review data outlives its purpose. A snapshot of a pending email or record update is customer data. Give it a retention window, and redact it after.

Reviewers drown, or nobody notices. One notification per pending action is noise when a workflow fans out, so bundle them. And put metrics and alerts on the review path, so a stuck queue shows up on a dashboard instead of in a customer’s inbox.

Result

• Human Review shipped from spec to production in about five weeks, covering the pause and resume path, approve and reject, apply-from-snapshot, reconciliation, notifications, monitoring and the reviewer’s inbox.
• Every action now separates resolving from applying, which is a better shape for the codebase whether or not review is on.

What I’d do differently

Make the resolve/apply split the default from the first action. It was the biggest piece of preparatory work, and it’s just good design. Every action that has to grow it later is a separate change to get right.

Treat workflow versioning as a habit, not an event. This was our first versioned workflow change, so the convention got invented alongside the feature. I’d write it down before the first change that needs it.

Design the review inbox as a general queue on day one. Ours was soon extended to carry other kinds of items. Building it generically from the start would have made those extensions smaller.

The views here are my own, not those of any employer, past or present.