Checkpointed agent jobs wasted 5x fewer model calls

Retrying a whole ten-step AI agent job after any failure wasted 4,143 model calls on 2,000 jobs, lost 75 jobs for good, and ran the email step twice for 273 jobs. Saving a checkpoint after each step cut the waste to 822 calls, lost no jobs, and cut the duplicate emails to 21. Adding an idempotency key to the email step brought the duplicates to zero. Each checkpoint write took 6.8 microseconds on SQLite (our lab, 2026).

This is the fourth post in the series on designing AI and SaaS products. It simulates 2,000 agent jobs with transient errors and worker crashes, compares three designs, and prices the wasted model calls with the per-request cost from the first post. It ends with a 25-line step runner and a guide to when a workflow engine is worth adopting.

What is a durable agent job?

A durable agent job is a multi-step task, usually several model calls and tool calls, whose progress survives crashes, deploys and restarts. It resumes from the last completed step instead of starting over.

Durable execution is the technique that makes that possible: record each step's result, and on restart skip the steps already recorded. Vercel's documentation describes workflows as durable code that can retry failed steps, wait for external events, and resume across crashes and deployments. Temporal's documentation describes recovery differently: after a crash it restarts the workflow code from the beginning and replays the event history, step by step, to rebuild the earlier state.

A checkpoint is a saved record that a step finished, ideally with its output. A side effect is any action with consequences outside your process, such as sending an email, charging a card or opening a ticket.

Agents need this more than web requests do. A web request takes milliseconds and costs almost nothing to repeat. An agent run takes seconds to minutes, makes paid model calls at every step, and often ends with an action that cannot be undone.

What did the lab simulate?

The lab runs 2,000 agent jobs. Each job has 10 steps, and each step makes one model call. Step 6 sends an email, standing in for any side effect.

Setting	Value
Jobs	2,000
Steps per job	10, one model call each
Side effect	Step 6 sends an email
Transient failure	3 percent of step attempts fail before doing any work
Worker crash	1 percent of attempts crash after the work, before it is recorded
Cost per model call	$0.0102, the cached Sonnet 5.5 request from the first post in this series
Store	SQLite in WAL mode, real writes
Runs	3 seeds per design, medians reported

The failure rates are my assumptions. A 3 percent transient rate stands for provider timeouts and overload errors, and a 1 percent crash rate stands for deploys, out-of-memory kills and function timeouts. Real rates depend on your stack, so the script takes them as constants.

A job with no waste makes exactly 20,000 model calls across the 2,000 jobs. Every call above that is waste.

Which three designs did we compare?

Whole-job retry runs all ten steps in one function. If any step fails or the worker crashes, the job starts again from step 1, up to three attempts. This is what you get from a plain queue with a retry count.

Checkpoint per step saves a row after each step. A failed step retries up to three times on its own. A crash returns the job to the queue, and the next worker resumes after the last saved step.

Checkpoint with an idempotent side effect adds one thing: the email step records its job ID first, and skips the send if the ID already exists.

Design	Jobs finished	Jobs lost	Model calls	Wasted calls	Wasted cost	Jobs with a duplicate email
Whole-job retry, 3 attempts	1,925	75	24,143	4,143	$42.26	273
Checkpoint per step	2,000	0	20,822	822	$8.38	21
Checkpoint plus idempotent email	2,000	0	20,822	822	$8.38	0

Checkpointing cut wasted calls from 4,143 to 822, a factor of five (our lab, 2026).

Why does whole-job retry fail so often?

Each attempt has to survive ten steps. With a 3 percent failure rate and a 1 percent crash rate, one step succeeds 96 percent of the time. Ten steps in a row succeed 0.96 to the tenth power of the time, which is 66.5 percent.

Three attempts raise that to 96.2 percent. The lab measured 96.3 percent, so 75 of 2,000 jobs were lost permanently. Those are users whose request vanished after the provider already charged for up to 30 model calls each.

Long agents make this worse quickly. A 20-step job succeeds in one attempt only 44 percent of the time at the same rates. The loss grows with every step you add, and agents keep getting longer.

The waste follows the same arithmetic. When step 9 fails, the job pays again for steps 1 to 8, and the provider bills every one of those calls.

Why did checkpointing alone still duplicate emails?

A checkpoint records that a step finished, and recording takes time. If the worker crashes after sending the email and before saving the checkpoint, the step looks unfinished. The next worker runs it again, and the customer gets a second email.

In the lab that happened to 21 of 2,000 jobs. Whole-job retry was far worse, with 273 jobs sending a second email, because any failure after step 6 replayed the entire job.

This is the same lesson as in the earlier post on usage metering: delivery is at-least-once, so effects must be idempotent. An idempotency key is an identifier sent with an action so that repeating the action has no additional effect.

Stripe documents this pattern for its API. Its guide to idempotent requests lets you retry a request safely with the same key. Use the same idea for your own effects, with a key such as job-id:step-number, and check it before acting.

For effects you cannot make idempotent, such as an email provider that has no dedupe, record the intent before sending and the result after. Then a replay can see that a send was started and look up its outcome before trying again.

What does a checkpoint cost?

A checkpoint write took 6.8 microseconds at the median and about 27 microseconds at the 99th percentile on SQLite in the lab. The store was in-process, so there was no network.

A hosted database costs more. In an earlier test of this blog's own test suite, one round trip from a laptop to a hosted MongoDB cluster took 263 milliseconds. Even at that price, ten checkpoints add 2.6 seconds to a job that makes ten model calls of several seconds each.

Compare that with the waste. The lab's whole-job design wasted $33.88 more than checkpointing per 2,000 jobs, about 1.7 cents per job, plus 75 lost jobs. A checkpoint that costs hundreds of milliseconds repays itself quickly when each step costs real money.

One caveat applies. The lab stored only the step number. A real agent must save each step's output too, such as the tool result and the model's response, so the next step can continue. Outputs can be large, so measure the row size before you pick a store.

Only the design with both a checkpoint and an idempotent effect had no customer-visible failure (our lab, 2026).

What does a step runner look like?

A step runner loads the saved checkpoints, skips completed steps, and saves after each new one. This version stores outputs and passes an idempotency key to every step.

// db.get(job) -> Map of step index to saved output; db.put(job, i, output) is idempotent
async function runJob(job, steps, db, { retries = 3 } = {}) {
  const saved = db.get(job);
  let state = saved.size ? saved.get(Math.max(...saved.keys())) : undefined;

  for (let i = 0; i < steps.length; i++) {
    if (saved.has(i)) continue;                       // finished before a crash
    const key = `${job}:${i}`;                        // idempotency key for side effects
    for (let attempt = 1; ; attempt++) {
      try { state = await steps[i](state, { key }); break; }
      catch (err) { if (attempt === retries) throw err; }
    }
    db.put(job, i, state);                            // checkpoint after the step
  }
  return state;
}

Run runJob from a queue consumer. If the process dies, the queue redelivers the job and the loop skips every saved step. The step that was running when the crash hit runs again, which is why its effects need the key.

Two rules keep this safe. Steps must produce the same effect when run twice with the same key. And a step must not depend on anything that changes between attempts, such as the current time, unless that value is saved in the checkpoint.

When should you adopt a workflow engine?

Adopt one when you have waits, branches or long-running steps, not just retries. The 25-line runner covers a linear job on a queue. It does not cover sleeping for three days, waiting for a human to approve, or fanning out to 50 parallel steps.

Vercel's documentation says its workflows can "pause for minutes or months, then resume from the exact point", and that they survive deployments and crashes through deterministic replays. It prices usage by events, data written and data retained. Temporal offers the same idea through event history replay, and runs for years in its own description.

The hidden cost of a replay-based engine is determinism. The workflow code runs again from the top after a crash, so it must make the same decisions each time. Calls to the clock, random numbers and network requests belong inside steps, where their results are saved.

Human approval is the clearest case for an engine. A job waiting for a manager's click cannot hold a worker for three hours. It must stop, free the worker, and resume when the click arrives, which is exactly what a durable sleep or hook does.

What happens to a job that never succeeds?

Cap the number of resumes and move the job aside. The lab allowed ten redeliveries per job, and none reached the cap. A real system also meets jobs that fail every time, such as a step that always throws on one malformed input.

Without a cap, that job loops forever and spends model calls on every pass. Send it to a dead-letter queue, a holding area for jobs that exceeded their retries, with its last checkpoint attached. A person can then fix the input and resume it from the saved step, instead of paying for the whole job again.

Add exponential backoff with jitter between step retries. If a provider is overloaded, 2,000 jobs retrying at the same instant make the overload worse. Random delays spread the load and give the provider time to recover.

What should you do first?

Start with the cheapest changes, in this order.

1. Count your steps and your failure rate. Multiply the per-step success rate across the whole job. If the result is under 90 percent, whole-job retries are already losing work.
2. Save a checkpoint after every paid step. The write is cheap, and the model call it protects is not.
3. Key every side effect. Use job-id:step and check it before sending, charging or creating anything.
4. Retry steps, not jobs. Retry a failed step three times with backoff, then return the job to the queue.
5. Alert on resumed jobs. A spike in resumes means a flaky step or a bad deploy.
6. Move to a workflow engine when you need waits or approvals. Not before.