Set up model fallbacks that fail safely

Every AI provider has outages, rate limits and bad days. A fallback list tells your gateway what to try next when the first choice fails, so customers see an answer instead of an error. The catch is that the backup may not be the model you tested. It can be the same model on a different host, and that host can give worse answers without returning an error.

How a fallback list works

Gateways put several AI providers behind one API. You name a first-choice model and a list of backups. OpenRouter takes a models array and tries each entry in order when the one before it returns an error. Rate limits, outages, a prompt that is too long and content-filter blocks can all trigger the next attempt. You pay the price of the model that answered, and the response names that model (OpenRouter).

Vercel's AI Gateway uses a models list as well, set in the gateway's provider options. A separate order setting ranks the hosts for each model. The gateway tries every listed host for one model before it moves to the next model. Each response carries a record of every attempt, with the host, the error and the time taken (Vercel).

Decide which errors should switch models

By default, OpenRouter treats any error as a reason to try the next model (OpenRouter). Most errors deserve a different response. Decide each one on purpose:

• Rate limit: wait and retry the same model first. Switching models on every brief limit means customers get the backup during each traffic spike.
• Outage or server error: move to the next host for the same model, then to the next model.
• Prompt too long: a model with a larger context window can help. So can trimming the prompt, which also costs less.
• Content-filter block: stop and think before you send the same request to a model with looser filters. The first model may have been right to decline.

A switch also has costs that do not show up as errors. A backup model may charge more per token than the first choice. It starts with an empty prompt cache, so the first calls after a switch pay the full input price. Answers may read differently, which matters for anything a customer sees.

The same model is not the same everywhere

Mo Moustafa runs Olly, an assistant inside iMessage that has handled over 18 million messages, about a third of them on open models through OpenRouter. His write-up of what he found is one of the most detailed public accounts of provider differences (Moustafa).

• On TAU-Bench Airline, a customer-service benchmark that tests tool use, DeepSeek V4 Flash scored 81.3% on DeepSeek's own API and 58.4% on DigitalOcean. Most hosts landed 5 to 7 points below the maker on tool use.
• Some hosts ignored attached images and still returned a success code.
• Some hosts accepted a reasoning-effort setting and then barely changed their behavior.
• Some responses came back marked as successful with no content at all.
• The declared precision of a host did not predict its quality.
• Pinning three hosts with fallbacks turned off led to an outage two weeks later, as each host degraded in turn.

Very different models in real life.

The lesson is not to avoid fallbacks. Olly went down when it removed them. The lesson is that a fallback is a quality decision as well as an uptime decision, and it needs its own test.

The gateway holds every key

A gateway stores the API keys for every provider it calls. That makes the gateway software itself worth protecting. On March 24, 2026, two releases of LiteLLM, a widely used open-source gateway, were published to the Python package index with code that stole credentials. The LiteLLM team deleted versions 1.82.7 and 1.82.8 and rotated its maintainer accounts. The team reported that users of its Docker image were not affected, because that image pins every dependency (LiteLLM).

How to build the chain

1. First choice: the model you tested, on its maker's own API.
2. Second: the same model on a second host you have tested yourself, from your production servers.
3. Third: a different model you have tested on the same tasks. Accept that answers may read differently.
4. Keep the list short. Each extra entry is a model you must test and watch.

Checks to run before you rely on it

• Run your own 20 to 50 real tasks against each entry in the chain, from production, not from a laptop. Moustafa found some hosts rate-limited his servers but not his own machine.
• Treat a success code with an empty answer as a failure, and retry.
• If you send images, confirm each host reads them.
• Pin the version of any gateway package you install, and update on purpose.
• Give the gateway keys with spending limits, so a leak has a ceiling.

Fallbacks handle failure. To choose a cheaper model on purpose, see Send easy work to a cheap model, and escalate on a check. To cut what each call costs, see Why an AI agent costs more with every turn.

Emeri did not run these tests. The findings come from the sources linked above.