Outage drill: a fallback saved 2,452 failed AI requests

When a model provider slowed down for 60 seconds in a simulation of 12,000 requests, a client with plain retries failed 18.5 percent of them and sent 1.62 provider calls for every user request. Adding a circuit breaker halved the calls and cut recovery from 21 seconds to 12. Adding a fallback answer turned all 2,452 failed requests into degraded answers, with zero errors shown to users. No client policy can create capacity the provider lacks, so the real choice is what your users see while the provider is slow (our simulation, 2026).

This is the sixth and last post in the series on designing AI and SaaS products. It runs a failure drill on a simulated AI product, compares four client policies, and ends with a checklist that turns each earlier post in the series into a drill you can run before launch.

What is a failure drill, and why simulate it?

A failure drill is a planned test where you cause a failure on purpose and measure how your system behaves. You run it before an incident does it for you.

Simulation is the cheapest drill for provider outages. You cannot make a real provider slow for 60 seconds, and you should not hammer one to find out. A model of the provider, with a few honest assumptions, shows the shape of the result.

Four terms matter here. Retry amplification is the extra load created when each failed request is sent again. A circuit breaker is a guard that stops sending requests to a failing dependency for a short time, then tests it with a few probe requests. Load shedding is rejecting work early to protect the rest of the system. Graceful degradation is serving a lower-quality answer, such as a cached reply or a smaller model, when the best path fails.

The OSDI 2022 paper "Metastable Failures in the Wild" studied 22 such failures from 11 organizations. A metastable failure is one where an overloaded system stays broken after the trigger ends. The authors report that load increase from retries was the most common sustaining effect.

What did the simulation model?

The simulation runs 300 seconds of virtual time. Users arrive at 40 requests per second, which is about 12,000 requests per run. Each request needs one model call.

Part of the model	Value
Provider capacity	120 concurrent calls, a full pool returns 429 at once
Normal call time	2 seconds on average
Capacity in normal times	60 requests per second, so the load is 67 percent of it
Incident	Seconds 60 to 120, call time rises to 20 seconds
Capacity during the incident	6 requests per second, 15 percent of demand
Abandoned calls	Keep their provider slot until they finish
Cost per call	$0.0102, the cached Sonnet 5.5 request from the first post
Runs	3 seeds per policy, medians reported

The last assumption is the one that matters most. A client that times out stops waiting, but a non-streaming provider call may keep running and holding its slot. Many providers cancel a call when the connection closes. If yours does, shorter timeouts waste less than they do here.

The provider's own limits are also real. Anthropic's rate limit documentation says limits are measured in requests, input tokens and output tokens per minute, and that exceeding one returns a 429 error. When the response carries a retry-after header, you wait that many seconds before retrying.

Which four client policies did we compare?

Each policy changes what the client does when a call fails or takes too long.

Policy	Timeout	Retries	Extra behavior
Naive	30 s	3, immediately	None
Backoff	8 s	2, with random delay up to 1 s then 2 s	30-second user deadline
Breaker	8 s	2, with backoff	Opens after 10 failures in the last 20 calls, stays open 10 s
Breaker with fallback	8 s	2, with backoff	Serves a degraded answer in 0.5 s when the breaker is open or retries run out

Policy	Full answers	Degraded answers	Failed	Provider calls per request	p95 latency	Seconds to recover	Wasted spend per run
Naive	81.5 percent	0	18.5 percent	1.62	4.6 s	21	$0.52
Backoff	78.5 percent	0	21.5 percent	1.46	3.7 s	23	$4.23
Breaker	79.8 percent	0	20.2 percent	0.83	3.6 s	12	$2.95
Breaker with fallback	79.8 percent	20.2 percent	0	0.83	3.6 s	12	$2.95

Only the fallback policy showed no failed requests to users (our simulation, 2026).

Recovery time is the number of seconds after the incident ended until 95 percent of requests in a 10-second window got a full answer. Wasted spend is the number of calls the provider started that never gave a user a full answer, priced at $0.0102 each.

Why can't retries fix an outage?

Retries add attempts, not capacity. During the incident the provider could finish about 6 calls per second, and users sent 40. A request that fails and retries only joins the queue for the same slots.

That is the floor. In a 60-second window, 2,400 requests arrive and about 360 can be served, so around 2,040 must fail, whatever the client does. Naive retries failed 2,236 requests in the whole run, within 10 percent of that floor.

Retries did cost something. The naive client sent 1.62 provider calls for each user request, because failed calls were sent again at once. After the incident ended, those retries kept the provider's slots busy, and the system needed 21 seconds to get back to normal.

The simulation did not show a permanent collapse. Normal load was only 67 percent of capacity, so once calls sped up again the backlog drained. A system running at 90 percent of capacity would not drain so easily. The OSDI paper's finding is that retries are the most common thing that keeps a system down, so test your own headroom.

Why did backoff and shorter timeouts do worse?

The backoff policy cut calls to 1.46 per request, and it answered fewer users, 78.5 percent against 81.5. It also wasted $4.23 per run, eight times the naive policy.

The cause is the 8-second timeout. During the incident, calls took about 20 seconds. The client gave up on them at 8 seconds and retried, but the abandoned calls kept running and kept their slots. The provider spent its scarce slots finishing answers nobody was waiting for.

The naive client's 30-second timeout waited long enough for many slow calls to finish. For a slow but working provider, patience delivered more answers than a tight timeout.

Two lessons follow. Set timeouts from the latency you observe, not from a guess. And cancel upstream when you abandon a call, if the provider supports it. A cancelled call frees its slot, and the waste in this simulation disappears.

What does a circuit breaker buy?

A breaker stops the client from sending calls it expects to fail. It cut provider calls from 1.62 to 0.83 per request, because rejected requests never reached the provider. Recovery fell from 21 seconds to 12.

The mechanism is simple. After 10 failures in the last 20 calls, the breaker opens and fails requests at once for 10 seconds. It then allows a few probe calls. If they succeed, normal traffic resumes.

The breaker halved the calls per request and cut recovery time by 9 seconds (our simulation, 2026).

The breaker did not add full answers, 79.8 percent against 81.5, because it gave up on some requests the naive client would eventually have served. In return the provider got room to recover, and users who would have waited 30 seconds for an error got a fast answer.

Recovery also needs care. Anthropic's documentation warns that a sharp increase in usage can cause 429 errors from acceleration limits, so it recommends increasing traffic gradually. After the breaker closes, ramp traffic instead of releasing the whole backlog at once.

What should the fallback be?

A fallback is any answer that is better than an error. It has to work when the main path does not, so it must not depend on the same provider slots.

Fallback	Quality	Cost
Cached answer for a repeated question	High for repeats	Nearly free
A smaller model, such as Haiku 5.5	Lower, varies by task	$0.00053 per request in the cost model of the first post
A different provider	Similar, if tested	Second contract and second set of prompts
Queue the request and notify the user	Full, but late	Cheap, adds a notification path
A plain message with a retry button	None	Free

A smaller model is the usual choice. Anthropic's tables give each model its own rate limit row, so a different model class draws on a different limit, although organization-wide limits still apply. A Forbes Technology Council column notes that a complex query sent to a frontier model can cost 50 to 100 times more, so a cheaper model is also the cheaper path in normal times.

Label the degraded answer. Tell the user the reply came from a faster, simpler path, and keep the original question so a full answer can follow once the provider recovers.

What does the wrapper look like?

This wrapper adds a breaker and a fallback around any model call. It is a sketch to adapt, not a library.

function withResilience(call, fallback, { window = 20, trip = 10, openMs = 10000 } = {}) {
  let recent = [], openUntil = 0;
  const record = (failed) => { recent.push(failed ? 1 : 0); if (recent.length > window) recent.shift(); };
  const tripped = () => recent.length === window && recent.reduce((a, b) => a + b) >= trip;

  return async function (input) {
    if (Date.now() < openUntil) return fallback(input);        // breaker open: fail fast
    try {
      const out = await call(input, { signal: AbortSignal.timeout(8000) }); // cancels upstream
      record(false);
      return out;
    } catch {
      record(true);
      if (tripped()) { openUntil = Date.now() + openMs; recent = []; }
      return fallback(input);
    }
  };
}

The AbortSignal.timeout line is the important one. It cancels the request when the timer fires, which frees the provider's slot if the provider honors cancellation. Add retries with random delay inside call if you want them.

How do the six posts become six drills?

Each earlier post found a failure that only appeared under stress. Turn each into a test that runs before every launch.

Drill	What to break	Pass condition	Post in the series
Cost drill	Replay a day of traffic through your cost model	Cost per request within 2 times your baseline	What an AI chat request really costs
Noisy tenant drill	One tenant sends 2,000 jobs at once	Other tenants' 95th percentile wait stays under 100 ms	A noisy tenant made everyone wait 9.6 seconds
Duplicate drill	Redeliver 3 percent of usage events and replay 500	Billed total equals the true total	Duplicate usage events overbilled an AI SaaS bill
Crash drill	Kill workers at random during jobs	No lost jobs, no duplicate side effects	Checkpointed agent jobs wasted 5x fewer model calls
Recall drill	Query with a tenant filter on your real embeddings	Recall at 10 above 0.95 for small and large tenants	pgvector plus a tenant filter found 4 percent of the right results
Outage drill	Slow the provider for 60 seconds	Zero errors shown, degraded answers labeled	This post

None of these needs production traffic. Each one is a script of under 200 lines, and each caught a real defect in this series: a 9.6-second stall, a $114.86 overbill, 75 lost jobs, a 3.6 percent recall.

Run them in a staging environment on a schedule, and fail the build when one regresses. A drill that runs once is a demo. A drill in your pipeline is a safeguard.

What should you do first?

Start with the outage drill, because it needs no new infrastructure.

1. Write down your provider's limits and your headroom. If normal load is above 70 percent of the limit, a retry storm can keep you down.
2. Set timeouts from measured latency. Use the 99th percentile of healthy calls, not a round number.
3. Cancel abandoned calls. Pass an abort signal to every model call.
4. Add a breaker. Open it after repeated failures and probe before closing.
5. Pick a fallback and label it. A cached answer or a smaller model is enough to start.
6. Ramp after recovery. Release queued work gradually.
7. Put all six drills in your pipeline. Review the results monthly.