What is model routing? Explained with an office front desk
Model routing sends each AI request to the model that suits it: simple jobs to a small, cheap model, hard ones to a large one. Think of an office front desk sorting enquiries.
- Author
Umair Mateen- Published
- Reading time
- 5 min
- Category
- AI
Google says its new Decisions API already powers model routing inside Gemini Notebook (our news post), and this week's news about Gemini 4 Carbon and Claude Fable 5.2 kept giving the same advice: send only the hard jobs to the expensive model. That habit has a name.
In one sentence: Model routing is a quick check that reads each request sent to an AI system and passes it to the model best suited to it, so simple jobs go to a small, cheap, fast model and only hard jobs go to a large, expensive one.
The everyday picture
Think of a busy office with a front desk. Every letter, call and visitor arrives there first.
Behind the desk sit two kinds of staff. Junior clerks are quick and cost little per hour. They handle address changes, simple forms and routine questions. One senior expert is slow and costs a lot per hour, but can untangle the difficult cases.
The person at the front desk reads each enquiry for a moment and decides who should take it. Nobody wants the senior expert spending the day on address changes. Nobody wants a junior clerk guessing at a hard case. Good sorting at the desk keeps the office fast, affordable and accurate.
The front desk (the router) reads every request. Simple ones go to the junior clerk (a small model), hard ones to the senior expert (a large model), and one answer goes back.
How it works, step by step
- A request arrives (an enquiry at the front desk). It could be a customer question, a document to sort or a step in an AI agent's task.
- The router reads it (the front desk takes a quick look). The router is often a small AI model itself. Microsoft says its model router weighs things like the request's complexity, the reasoning it needs and the type of task.
- The router predicts which model will do well enough (who can handle this case). Amazon's router predicts the answer quality each model would give and compares it with a default "fallback" model.
- Easy requests go to a small model (a junior clerk). It answers quickly and costs less per request.
- Hard requests go to a large model (the senior expert). It costs more, but is more likely to get a complex job right.
- One answer comes back (the reply leaves the office). The person who asked usually never sees which model did the work.
Try it · What would you do?
Where should the front desk send this request?
Your support assistant uses a router with a small model and a large model. A customer writes a long, angry email about a refund that was promised twice and never paid.
Pick what you would do. You can change your mind.
Where the picture stops
- A front desk can stop and ask a question. A router makes a prediction in a split second, and it can be wrong. Unless you build in a second check, a hard question sent to a small model simply gets a weaker answer.
- Office staff share the same files. AI models don't always. Microsoft notes that with its router, the usable context window (how much text a model can read at once) is limited by the smallest model in the mix.
- A good receptionist learns your office over time. Ready-made routers may not. Amazon says its router can't adjust its decisions based on your own application's results, and is optimised for English prompts only.
Why it matters for your business
- Cost. If most of your requests are routine, sending them to a cheaper model can cut your bill. In the RouteLLM research paper, the authors report costs cut by over two times in certain cases, without lower answer quality on their benchmarks.
- Speed. Small models usually answer faster, so customers wait less for simple answers.
- Quality where it counts. The expensive model is kept free for the cases where a mistake would cost you.
- Risk. A router is one more thing that can go wrong. Test it on your own real requests, and check how many hard cases slip through to the small model.
An example
Imagine a 30-person online furniture shop with an AI assistant answering customer emails. Most emails ask where an order is or how to return an item. A few are complaints about damaged goods that need careful, fair replies.
The shop adds a router. Order and return questions go to a small model. Complaints go to a large model, and its draft goes to a staff member before sending. The shop checks a sample of routed emails each week to see whether any complaint went to the wrong place.
Frequently asked questions
Is model routing the same as an AI agent?
No. An agent works through a task in many steps. A router only decides which model handles each request. Agents often use routing so that routine steps run on a cheaper model.
Do I need a special product to route between models?
No. A simple version is a few rules in your own software, such as "complaints go to the large model". Cloud providers, including Amazon and Microsoft, also offer ready-made routers that decide for you.
Does routing always save money?
No. If most of your requests are hard, nearly everything still goes to the large model. Savings depend on how many of your jobs are simple, so measure that first.
Can I see which model answered?
Often, yes. Amazon says its router's response includes information about the model it used. Check what your provider shows before you rely on it for audits.
Related explainers
The DevDrift helps teams choose and route AI models with cost controls built in: see how.
Sources
Could you explain this to a colleague now?
Mark it and posts that mention it will show a tick. Saved in this browser only.