Intelligent routing

Most teams building on LLMs are over-provisioned: they picked a frontier model once, pointed every request at it, and never went back. That is a rational choice. You cannot tell in advance which requests need the expensive model, so you pay frontier prices for all of them, including the majority a far cheaper model would answer just as well.

AICP's auto routing removes that trade-off. For every prompt it predicts the cheapest class of model that can still answer it well, and sends it there. Hard prompts still reach a frontier model; easy ones stop costing like they do.

This page explains how that decision is made. For what it delivers and how we measured it, see Routing performance. For the configuration surface, see Routing.

Two stages, two different questions

A routing decision answers two questions, and AICP keeps them separate because they have different answers for different customers.

   your prompt
        │
        ▼
  ┌───────────────────────────────────────────┐
  │ 1 · Tier prediction                       │   "How capable does the
  │    prompt → small | medium | large        │    model need to be?"
  └───────────────────┬───────────────────────┘
                      ▼
  ┌───────────────────────────────────────────┐
  │ 2 · Model selection                       │   "Given that floor, which
  │    governance filter → strategy → winner  │    model wins under your
  └───────────────────┬───────────────────────┘    policy?"
                      ▼
              provider call
                      │
                      ▼
        response + X-Router-* headers
              + auditable decision record

Stage 1 is a property of the prompt. Stage 2 is a property of your organisation: your enabled providers, your regions, your budget ceilings, your strategy. Splitting them means your compliance rules never have to be encoded into a model, and the model never has to be retrained because you added a provider.

Stage 1: predicting the tier

The router scores each prompt across the three tiers and routes to the highest-scoring one. That is the whole rule:

scores = router(prompt)          # one score per tier
route to the tier with the highest score

There is deliberately no threshold on top of it. The cost/quality trade-off was already priced when the router was trained — the training objective rewarded a tier for answering the prompt well and penalised it for what it costs, so the router's own preference is the routing decision. Applying a confidence threshold afterwards would override a trained judgement with a second, separately-tuned opinion about the same question, and double-count the trade.

Measured tier mix on the held-out evaluation set

21% small · 17% medium · 62% large

Read this together with the cost: routing 62% of requests to the frontier tier still lands at 41% of always-frontier spend, retaining 95% of its quality, because the 38% that go elsewhere are the cheapest requests to serve and the frontier tier is priced far above the others. Your mix will differ — it is a property of your traffic, not a setting.

Two behaviours worth knowing:

• Degenerate input never fails a request. Empty, whitespace-only or non-text prompts short-circuit to the cheapest tier and return a normal decision rather than an error.
• A router outage degrades to your strategy, not to an error. The tier call carries a 2-second deadline; if it times out or the service is unreachable, the request proceeds without a predicted tier and your configured strategy selects from the full catalogue. Routing never fails a request because the classifier is unavailable.

Prompt text is redacted before it reaches the router, and the router makes no network calls of its own. No LLM sits in the routing hot path.

What the router actually looks at

The router is a linear classifier over a multitude of hand-crafted features computed directly from the request text — measurements of its size, structure, composition and phrasing, read straight off the string. There is no embedding model or semantic vector anywhere in the decision.

That is a deliberate design choice, not a simplification. Most routers of this kind embed the prompt first, which means a second model inference before the real one — a model to download, a dependency to ship, and its own latency on every request. Ours does not: the features are arithmetic so there is no second inference in the hot path at any point.

The consequence is that routing overhead is sub-millisecond — measured at 0.4 ms median, 0.9 ms at p95, in-process, on a call that takes seconds. Nothing is billed for it and no tokens are consumed.

note

A tier is a quality floor, not a fixed model. Every model the router can select in a tier has been measured to clear that tier's floor, which is why the router can optimise freely inside a tier without putting quality at risk. Models that clear no tier's floor, or whose measurement was inconclusive, are marked provisional and excluded from automatic routing — you can still name them explicitly. How those floors were established is covered in Routing performance.

The trade-off is set at training time

One number controls how aggressively auto saves money. It is λ, the exchange rate between a point of quality and a unit of cost, and it is fixed when the router is trained rather than adjusted per request.

λ	Behaviour	Result
Higher	Cost weighs more against quality; cheaper tiers win more often	Cheapest, most risk
Lower	Quality dominates; more traffic escalates	Safest, dearest

Because λ changes the training target rather than a threshold, each setting is its own checkpoint and its own measurement — not points interpolated along one curve. Two are published, both measured against always-frontier on held-out data: a cost-leaning setting at ~22% of frontier cost for ~90% of its quality, and the deployed default at ~41% for ~95%. See Routing performance for the comparison and the baselines it should be judged against.

How the router is trained

The router is trained to predict the best-value tier, not difficulty. That distinction is the whole design, and it shows up in how the training labels are made.

Labels come from outcomes, priced. Every training prompt was answered by a reference model from each tier, and each answer graded independently against a correctness threshold. When two tiers answer a prompt equally well, the cheaper one wins by construction, which is the behaviour a router should learn and which a plain tier classifier never sees.

Prompts are weighted by what the decision is worth. Each training row carries the regret between the best tier and the second best. A prompt every tier answers identically is free to get wrong and does not pull the boundary around; one where the wrong tier costs real money does. Weighting them equally is how a classifier ends up optimising accuracy on decisions nobody is paid for.

The target is therefore "which tier is the best buy here", not "how hard does this look". The two come apart constantly: a long prompt full of code can be trivially answerable, and a twelve-word question can need a frontier model.

Training is offline and self-contained. Features are computed from text with no model calls at all. Nothing about training touches the serving path, and no customer traffic is used.

A slice is held back, grouped. The evaluation split is drawn on whole scenario groups, not individual prompts. The corpus contains multiple rewrites of the same underlying scenario in different writing styles, and splitting on prompts would put near-duplicates on both sides of the line — which reads as skill and is not. A locked test portion is never fitted on and is read once.

The model is chosen by the routing outcome, not by classification accuracy. This matters more than it sounds. On traffic where most prompts are answerable by the cheapest tier, a classifier that always predicts "cheapest" scores impressively on accuracy while routing nothing useful. Candidate models are judged on the cost and quality they actually realise under the rule that runs in production, against the full set of baselines — every tier used for everything, and a perfect-knowledge oracle.

The result ships as a versioned artefact recording its feature set, its routing rule, its λ and the tier reference models it was labelled against. The service refuses to serve a checkpoint under a feature set or rule it was not trained for. It is retrained deliberately — when the tier reference models change, or the catalogue moves enough to shift what a tier represents — not on a schedule, so a given release routes the same way every time.

Stage 2: choosing the model

Once a tier is fixed, AICP expands it into every (model, provider) pair you could legitimately be served by, then narrows and ranks.

First, a hard governance filter. This runs before selection and applies identically whether the tier came from the router or you named a model yourself. A candidate is excluded if the provider is disabled, failing health checks, not authorised for your organisation, or in a provider category you have not allowed; if the model is unavailable; if the provider is outside your permitted regions; if it lacks a capability the request needs; or if it would breach your configured latency or per-request cost ceiling. Exclusions are recorded with their reason, so "why wasn't this provider used?" is always answerable.

Then your strategy ranks what survives. cheapest, fastest and highest_quality rank on a single axis. balanced, the default, normalises cost, latency and quality across the surviving candidates and combines them under weights you control (by default weighted toward cost, then latency and quality evenly). Your preferred-provider order applies as a bounded nudge on top, so a preference can break a close call but cannot override a large difference. The five strategies are documented in Routing.

Models too close to rank are treated as tied. Quality figures carry a measurement resolution, and candidates inside it are collapsed into one band and separated on cost instead. This is why highest_quality often returns the same model as cheapest within a tier — the quality difference it would be trading against is frequently not real, whereas the price difference is.

Latency comes from your traffic, not a benchmark

Every model starts with a benchmark-derived latency estimate. That estimate is a cold-start value only. Once a model has accumulated enough recent requests on your platform traffic, the observed figure replaces it. Routing then reflects how each model actually performs for you: your regions, your prompt shapes, that provider's real load.

Two details make the number trustworthy:

• It is normalised per output token. Raw end-to-end latency is dominated by how long the answer was, so a model that happens to receive short prompts would otherwise look faster than it is. Normalising removes that bias and makes streaming and non-streaming calls comparable.
• It never falls back to infinity. A model with too little recent traffic reverts to its estimate rather than being ranked last, so low-volume models stay in genuine contention instead of being quietly starved of the traffic they would need to prove themselves.

The switchover is continuous, not a one-way latch: a model whose traffic dries up drops back to its estimate automatically.

Using auto with streaming, tools and multi-turn

	Behaviour
Streaming	Supported. Routing is resolved before the stream opens, so a routing failure still returns a normal JSON error rather than a half-written stream.
Multi-turn	The router reads the most recent turns of the conversation, not just the latest message, so a short follow-up carries the context it depends on. Very long conversations are truncated from the front, keeping the tail — the current request is never the part that gets dropped. The tier is decided per request, so a conversation can move between tiers as it gets harder or easier.
Tool calling	Supported. A request carrying tools is routed only to providers that declare function calling. The check is at the provider level: it does not rank how reliably an individual model invokes a tool once offered one, so on quality-sensitive tool traffic a higher tier or an explicit model is the safer choice.
Structured output	response_format is not supported yet. The field is ignored rather than honoured, and routing does not take it into account. To get JSON reliably today, name a model you know supports it and instruct the model in the prompt.

When things go wrong

Situation	What AICP does
No model in the predicted tier is eligible	Escalates upward to a stronger tier, never down to a weaker model. The response carries X-Router-Tier-Escalated.
Nothing is eligible at any tier	Returns a clear no_available_model error rather than serving something you excluded.
Your policy excludes every provider	Applies your fallbackBehavior: fail, or relax to commercial / global / cheapest. The relaxation is recorded on the decision.
A provider rate-limits, times out, goes down, rejects credentials, or no longer knows the model	Excludes that provider and re-routes to the next candidate in the same tier, up to three attempts. Applies in auto mode only.
A request is malformed, or your policy rejects it	Surfaced immediately. These are true at every provider, so retrying would spend latency to fail identically and bury the real error.
Every provider is rate-limited	Returns 429 rather than hanging.
The tier router is unavailable	Falls back to strategy routing over the full catalogue. The request is served, not failed.

Two limits worth knowing up front. Cross-provider retry applies to non-streaming requests in auto mode. A streaming response that fails after the first token has already been sent is surfaced as a stream error rather than silently re-routed. And retry is a failover mechanism, not a load balancer: AICP reacts to a failure rather than tracking per-provider rate-limit budgets in advance.

Everything is auditable

Routing that you cannot inspect is routing you cannot trust. Every request carries its decision back to you in headers:

Header	Meaning
X-Router-Tier	The tier chosen for the prompt (0 small · 1 medium · 2 large)
X-Router-Confidence	The router's score for the tier it chose
X-Router-Model	The model actually served
X-Router-Estimated-Saving	Estimated fraction saved versus always using the large tier
X-Router-Tier-Escalated	Present only when the request was escalated

And every decision is persisted with its full candidate set, the exclusions and their reasons, the strategy applied and the resulting cost, all retrievable through routing history or the dashboard's request explorer.

See also

• Routing: configuring strategies, fallback and aliases
• Routing performance: how the router was evaluated, and what it delivers
• Routing in the JavaScript SDK · in the Python SDK