A year into using language models, most companies have the same three problems. There are keys for five providers in a dozen places. Nobody can say what last month's spend bought. And every team has quietly picked the strongest model for everything, because nobody was ever told off for an answer that was too good.
The fix for all three is the same piece of plumbing: one gateway that every call goes through.
One door
In Camaan every model call, from an agent, a workflow, an app someone built or an outside system holding a key, reaches the model through the same gateway. It speaks the wire format most tools already use, so nothing has to be rewritten to use it.
Because there is one door, there is one place to decide things:
- Who may use which model. An organisation can restrict its people to approved models. A business unit inherits that from its parent and cannot switch it off.
- How much they may spend. Keys carry budgets and limits.
- What gets recorded. Each call is written down once, with its tokens, its cost, where it came from and what it was for.
That last one matters more than it sounds. When we recorded calls only at the public endpoint, the log described that endpoint and nothing else. Moving the recording down into the client that every caller uses made the picture complete, and it made the gaps visible: a call from a path that does not yet say what it is shows up as "unknown" rather than disappearing.
Routing by difficulty
Once every call passes one point, you can choose the model there instead of in the caller. We sort requests into four tiers, from simple through complex and reasoning to agentic, and each tier maps to a model and its fallbacks.
The router that does the sorting is deliberately small. It is a short list of readable rules in front of a compact classifier that ships inside the product. It needs no other model to run and nothing to download, so it works the same in a cloud and in a data centre with no route out.
On 80 prompts it had never seen, that router picked the right tier 92.5% of the time. The classifier on its own managed 76.2%.
What we got wrong first
The first classifier was trained on labels that meant "the cheapest tier that coped". That sounds like the right target. It is really a statement about which models were good in the year the data was collected. The classifier learned it faithfully and sent a request to prove a theorem to the cheapest tier.
Rules in front of it fixed the symptom. Relabelling the data against a written definition of each tier fixed the cause. The lesson is an old one: write down what the label means before you train anything on it.
One more choice is worth copying. When the router is unsure between two tiers, it takes the higher one. A wrong step up costs a little money. A wrong step down costs a poor answer, and poor answers are how people stop trusting the system.
What it saves
We priced the same 80 prompts under different policies, using illustrative prices. Sending everything to the strongest tier cost about three times as much as a perfect router would. The router we ship came within 1.4% of the perfect one.
There was a cheaper option. The classifier alone cost about a third less than perfect routing, and it got there by under-serving more than a fifth of the requests. That is the trap in every cost dashboard: a saving that comes from giving people worse answers looks exactly like efficiency until someone checks the answers.
What the door costs
A gateway adds a hop, so we measured it. On a production-style build the gateway added about 14.5 milliseconds at the median for a single caller. A model takes hundreds of milliseconds to start answering, so that is noise. Under heavy concurrency the added time grows, and past a configured limit the gateway refuses new work quickly instead of letting a queue build. We would rather tell a caller "not now" in a few milliseconds than let everyone wait.
Where to start
You do not need a router on day one. Start with the door. Put every call through one gateway, record each one with its cost and its purpose, and look at a month of it. The routing policy tends to write itself once you can see what people are really asking.