Send easy work to a cheap model, and escalate on a check

The cheapest model on a price list can handle many requests on its own. A cascade sends each request to that model first and passes it to a stronger model only when needed. Done well, it cuts the bill without lowering quality. Done badly, it pays for two models and gets the quality of the weaker one. The difference is what decides the hand-off.

What the research found

The idea has a published track record. In 2023, Lingjiao Chen, Matei Zaharia and James Zou described FrugalGPT, a cascade that learns which models to try for which queries. They reported matching the best single model they tested with up to 98% lower cost (arXiv). The authors also noted that API prices for comparable tasks differed by up to two orders of magnitude.

In 2024, the LMSYS team released RouteLLM. It trained routers that choose between one strong model and one weak model for each query. The team reported cost reductions of over 85% on MT Bench, 45% on MMLU and 35% on GSM8K, while keeping 95% of the strong model's performance (LMSYS). Those savings varied widely by task. Routing helped most where many questions were easy.

A cascade that worked in production

Andrea Luzzardi described one at Mendral, a company whose agent investigates failed software builds (Mendral). A small model, Claude Haiku, looks at each failure first. It answers one narrow question: is this problem already known? If it is, the work stops there. If not, or if the small model is unsure, the failure goes to Claude Opus for a full investigation.

• In one week the system saw about 4,000 build failures. 3,187 were known problems coming back. 818 were new.
• A match by the small model costs about 25 times less than a full investigation.
• The small model handles about 65% of all input tokens for about 36% of the spend.
• Without the two-model setup, the team reports that the daily bill more than doubles.

When in doubt, it escalates.

The design has three features worth copying. The small model gets a yes-or-no question, not an open task. It checks its answer against a lookup of known errors, so the hand-off rests on evidence. Uncertainty sends the work up, never down.

A router that was switched off

Not every attempt pays. Bruno Perez wrote about why Manifest, a company that runs an AI gateway, shut down its own model router (Manifest). The router sorted each request into one of four tiers and sent it to a model to match. It ran for four months across 7,000 users. The team gave four reasons for removing it:

• The prompt does not show how hard the task is. It is only the trigger for work the router cannot see.
• Switching models works against prompt caching, which is often the larger saving.
• Changing models within a session made the output less consistent.
• The extra layer made testing, prompts and monitoring harder, and could cost more than it saved.

The contrast with Mendral is useful. Manifest guessed difficulty from the request. Mendral checked a fact about the request. A cascade earns its keep when a cheap, reliable check decides the hand-off.

Try one model at lower effort first

Before you build a cascade, test a simpler option. Many current models let you set how much they think, and a strong model at low effort can be cheaper than a weak model at full effort. Anthropic published results on this (Anthropic):

• On one of Anthropic's benchmarks, Claude Opus 5.5 at its default setting matched Claude Fable 5.1 (92.8% against 92.3%) at about a fifth of the cost per solved task.
• On the SWE-bench Pro coding benchmark, running Claude Opus 5.5 at low effort, then re-running only the failures at high effort, reached about 97% for about $0.17 per task. Running everything at high effort reached 95.3% for about $0.29.
• Pairing a cheap model with a stronger adviser it could consult often bought little. In two of Anthropic's tests the cheaper model never asked for advice on any of 198 questions.

The second result is a cascade inside one model. It works because a failed test is a clear signal. Price matters too. Claude Haiku 5.5 costs $0.10 per million input tokens for prompts up to 100,000 tokens, against $4 for Claude Opus 5.5 (Anthropic). A gap that wide gives a cascade room to save, if the check is sound.

How to set one up

1. Pick one high-volume job with a clear check. Examples: a reply that must match a known answer, data that must pass a validator, code that must pass its tests, a ticket that must fit one of your categories.
2. Give the cheap model a narrow question with a fixed set of answers, including "unsure".
3. Send anything marked unsure, and anything that fails the check, to the stronger model.
4. Switch models between tasks, not partway through a conversation. A mid-conversation switch starts the cache over.
5. Log which model handled each request and whether the result passed.
6. Compare cost per completed task, not cost per request. A cheap request that needs a retry is not cheap.

Routing decides which model gets a request. Two related guides cover the rest of the bill: Why an AI agent costs more with every turn and Set up model fallbacks that fail safely.

Emeri did not run these tests. The figures come from the sources linked above. Emeri's certification grading runs on Claude models.