Handle AI failures and retries gracefully

A model call is a network call to a busy service that sometimes says no, says nothing, or says something malformed. Treat every call as fallible: wrap it in a bounded retry with backoffWaiting a little longer before each retry so you do not hammer a struggling service, validate what comes back, fall back to a simpler path when it keeps failing, and surface a clear message instead of a stack trace. Do that, and one bad call degrades a feature instead of crashing the system.

What are the ways a model call can fail?

You cannot design for failure you have not named. Model calls fail in four distinct ways, and each wants a different response.

The common mistake is treating all four the same: retrying a refusal forever, or crashing on a transient blip that a single retry would have cleared. Sort the failure first, then act.

How do I tell a transient error from a permanent one?

A transientA temporary failure that is likely to clear if you try again error is worth retrying because the cause is temporary: the service was momentarily busy, a request timed out, or you briefly exceeded a rate limit. A permanent error is not, because retrying the same bad request changes nothing.

Anthropic's API signals this in the HTTP status. A 429 is a rate limit on your account, and a 529 means the service itself is overloaded. Both are transient and retryable. A 429 response carries a retry-after header telling you how many seconds to wait, so read it and honour it before falling back to your own timing. By contrast a 400 (bad request), 401 (auth), or 403 (forbidden) is permanent: the request is wrong, so fix the request rather than repeat it.

If you are unsure whether a given error is retryable, default to not retrying. A few unnecessary failures are cheaper than a retry storm against a service that will never succeed.

Retry with backoff and jitter

When you do retry, do not retry immediately and do not retry at a fixed interval. Use exponential backoff: wait a little after the first failure, then roughly double the wait each time, up to a cap. This gives a struggling service room to recover instead of being hammered.

Add jitterA small random offset added to each backoff delay so many clients do not all retry at the same instant on top. Without it, every client that failed at the same moment retries at the same moment, recreating the exact spike that caused the overload. This is the thundering herd problem. A random offset of plus or minus thirty percent on each delay spreads the retries out.

Three rules keep backoff honest:

• Honour retry-after first. If the response tells you how long to wait, that beats any number you computed.
• Cap the attempts. Three to five retries is plenty. Retrying forever turns a brief outage into a hung request.
• Cap the total time. A per-call budget (say thirty seconds across all attempts) stops a slow failure from blocking everything behind it.

You rarely have to build this by hand. The official SDKs retry 429, 529, and connection errors with exponential backoff automatically, and you tune the count with a max_retries setting. When you drive Claude through the Claude Code CLI or the Claude Agent SDK, transient retries are handled inside the tool. Your job is the layer above: deciding what happens when the retries are used up.

Retry only what is safe to repeat

There is a trap in retries that is easy to miss. If a call has a side effect, charging a card, sending an email, writing a row, a retry can run that side effect twice. The first attempt may have succeeded on the server and only the response got lost on the way back, so retrying duplicates the action.

The fix is idempotencyA request you can safely repeat: running it twice has the same effect as running it once. Attach a stable key to the operation, the same key on every retry, so the server recognises the repeat and returns the original result instead of acting again. Many APIs support an idempotency key on writes for exactly this reason; check your provider's docs for whether and how it is supported. As a rule, retry reads freely, but only retry writes when they are idempotent. If a step is not safe to repeat, do not put it inside the retry loop; do the retryable work first and commit the side effect once at the end.Common gotchaThe cruel version of this is wrapping a tool call that charges a card or sends an email in the same retry block as the model call.A timeout often means the write already landed and only the response was lost, so the retry runs it a second time and you double-charge a real customer. Keep side-effecting writes out of the retry loop, or gate them behind an idempotency key the server can dedupe.

How do I set a timeout on a model call?

A call that never returns is worse than one that fails, because it ties up the request behind it with no signal. Set a timeout on every model call so a stuck request becomes a clean, classifiable failure you can retry or fall back from.

Two numbers matter. A per-attempt timeout bounds how long any single try can hang. A total deadline bounds the whole operation including all retries, so backoff cannot quietly stretch a thirty-second budget into two minutes. When the total deadline is near, stop retrying even if attempts remain; there is no point starting a try you cannot finish in time.

Streaming responses change the calculation. With streaming, you get tokens as they are produced, so the right timeout is on the gap between chunks, not on the whole response. A long answer that keeps streaming is healthy; a stream that goes silent for ten seconds is stuck. Watch the inter-chunk gap and abort when it stalls.Common gotchaPutting one whole-response timeout on a stream backfires both ways: a short cap kills a healthy long answer mid-thought, and a generous one lets a stream that has gone silent hang for the full duration.Time the gap between chunks instead, and reset the clock on every token you receive.

How do I validate model output?

A 200 response is not a correct response. The model can return valid prose that is the wrong shape, malformed JSON, a value outside the allowed set, or an answer to a different question. If a downstream step parses that output, garbage propagates silently until something far away breaks.

Validate at the boundary, the moment output arrives and before anything else touches it. For structured output, the strongest tool is a schema. Ask for a known shape, then check the result against it.

{
  "name": "extract_order",
  "input_schema": {
    "type": "object",
    "properties": {
      "order_id": { "type": "string" },
      "total": { "type": "number" },
      "status": {
        "type": "string",
        "enum": ["paid", "pending", "refunded"]
      }
    },
    "required": ["order_id", "total", "status"]
  }
}

A tool schema like this does double duty: it tells the model exactly what to produce, and it gives you a precise definition to validate against. If the output fails the schema, you have caught a bad response at the boundary instead of three steps later. For more on shaping tool definitions as config, see tool use and function calling.

When validation fails, you have two moves. Re-ask once with the error fed back in (the model often fixes its own mistake when told what was wrong), and if it fails again, treat it as a bad output and fall back. Do not loop on re-asking forever; cap it like any other retry.

How do I handle a model refusal?

A refusal is the model doing its job: declining a request it judges unsafe, out of scope, or impossible. Retrying it unchanged wastes calls. Handle it as its own case.

Often a refusal means the request was ambiguous or under-specified. Reframing with more context, a narrower scope, or an explicit statement of intent gets a useful answer. When a refusal is correct and persistent, route to a human rather than fighting it. A clear "this needs a person, here is why" is a better outcome than ten retries that all decline.

Fall back instead of falling over

Retries buy time against transient trouble, but some failures persist. A fallbackA simpler or alternative path you take when the primary one keeps failing is the alternative you take when the primary path is used up, so the user gets a degraded answer instead of no answer.

Fallbacks form a ladder, best option first, each rung simpler and more reliable than the last:

• Retry the same call for a transient blip.
• Try a cheaper or different model if one is overloaded. The official guidance for a persistent 529 is to fail over rather than keep retrying.
• Return a cached or last-known-good result when freshness can be traded for availability. See prompt caching for the related idea of reusing stable context.
• Degrade the feature. Show search results without the AI summary, rather than showing nothing.
• Hand off to a human or a deterministic rule when nothing else works.

The point of the ladder is that the system always has a next rung. A feature that quietly drops to search-without-summary is still working. A feature that throws a 500 is not.

Stop one failure from spreading

In a system with many calls, a single failing dependency can drag everything down: requests pile up waiting on the slow service, threads exhaust, and the whole app stalls. Two patterns contain the blast radius.

A circuit breakerA switch that stops calling a failing service for a cooldown period after too many failures, so it can recover watches the failure rate of a dependency. After too many failures it "opens" and short-circuits further calls for a cooldown, returning the fallback immediately instead of waiting on a service you already know is down. After the cooldown it lets one trial call through; if that succeeds, it closes again. This stops you from spending your whole retry budget on a service that is fully down.

A budgetA hard ceiling on cost or work for a single run, after which it stops caps cost and effort per request. Set a maximum number of model calls, tokens, and wall-clock time for a single operation, and stop when any ceiling is hit. This pairs with the guardrails that fail closed when a ceiling is breached. Budgets matter most in agent loops, where a confused agent can otherwise retry and re-plan until it has burned far more than the task was worth.

For agentic runs specifically, add durable checkpoints: save state after each meaningful step so a failure resumes from the last good point instead of restarting from scratch. The patterns here line up with the multi-agent architectures production checklist.

This table sums up the decision you make on every failure:

Watch the failure rate, not just the failures

Individual failures are normal and your handling absorbs them. The signal worth alerting on is the rate. A slow climb in 529 responses, retry counts, or fallback hits tells you a dependency is degrading before it falls over completely. Track three numbers per feature: the share of calls that fail, the share that exhaust retries, and the share that land on a fallback. When the fallback rate rises, users are getting the degraded experience even though nothing is throwing, so it is the early warning that a healthy-looking system is quietly slipping.

A worked example

Tie the pieces together with a concrete feature. An online store has an endpoint at src/orders/summarize.ts that asks a model to turn a customer's order history into a short, friendly summary shown on their account page. It is non-critical: nice to have, not load-bearing. Here is the full path under failure.

The call goes out with a per-attempt timeout of ten seconds and a total deadline of twenty-five. The first attempt returns a 529, so the wrapper waits about one second plus jitter and tries again. The second attempt times out. The wrapper waits about two seconds plus jitter and tries a third time, which succeeds.

Now validation runs. The summary is supposed to come back as structured output with a headline and a bullets array, checked against a schema. This time the bullets field is missing. The wrapper re-asks once with the validation error included; the model returns a valid shape and the response passes.

Suppose instead the retries had all failed. The wrapper drops down the fallback ladder: it serves the last summary it cached for this customer, stamped with a quiet "updated earlier today" note. If there is no cached summary, the page shows the order list with no AI summary at all. The customer still sees their orders. Nothing on the screen says "error", because from the customer's point of view nothing is broken.

Throughout, every failed attempt is logged with its error type, the customer id, the retry count, and which fallback fired. When someone reports "my summary looks stale", that trace says exactly why.

Fail with a clear message

When you have exhausted retries and fallbacks, the last thing the user sees matters. A raw stack trace or a generic "something went wrong" tells them nothing and tells you nothing when they report it.

A good failure message has three parts: what happened in plain language, what the user can do now, and a quiet identifier for support. Keep the internal detail in your logs, not on the screen.

Behind the message, log the real cause: the error type, the request that triggered it, how many retries ran, and which fallback fired. That trace is what turns a vague bug report into a fix. Agents are non-deterministic between runs, so a failure you cannot reproduce is one you can only debug from logs.

Have Claude harden a call site

You do not have to wire all of this by hand. Point Claude at a fragile call site and have it apply the full pattern. A precise prompt gets a precise result.

Ask Claude to write tests for each failure path, not just the happy path. A test that forces a timeout, a malformed response, and a refusal proves the handling works before a real user finds the gap. See test and debug your agent for the workflow.

The shape of resilient calls

Every reliable model call follows the same order. Classify the failure, retry the transient ones with backoff and jitter, validate the output at the boundary, fall back down a ladder when retries run out, contain the damage with a circuit breaker and budgets, and surface a clear message while logging the real cause. None of it is exotic. It is the same discipline you would apply to any unreliable dependency, applied to a dependency that happens to be a model.