Routing performance
Intelligent routing describes how a request reaches a model. This page describes how the inputs to that decision are established: where each model's quality figure comes from, how precise it is, and how often it is refreshed.
Figures on this page were measured in August 2026. They are re-measured as the catalogue changes; see Re-measurement for what is refreshed and when.
Model quality is measured, not asserted
Routing is only as good as what it believes about the models it routes to. Quality figures in a model catalogue are usually copied from published benchmarks, which measure something other than the traffic you send and go stale silently. AICP measures its own.
Every model in the catalogue is run through a fixed instrument. This is a measurement instrument. It scores models so the tiers mean something; it is not what the router was trained on. For that, see how the router is trained.
| Component | What it fixes |
|---|---|
| Versioned prompt corpus | A frozen set spanning reasoning, mathematics, knowledge, open-ended and retrieval-style tasks. Objectively checkable items are graded by exact match; the rest by an independent LLM grader. |
| Frozen grader specification | Model, temperature, rubric and output contract, fingerprinted, so a change to grading is detectable rather than silent. |
| Per-model generation parameters | A shared configuration measures our settings rather than the model. Reasoning models get output budgets sized for reasoning overhead; models that reject a shared temperature get their own. |
| Two anchor models | Every score sits on one comparable axis instead of being calibrated separately against a moving benchmark. |
| Snapshot-keyed ledger | A record is keyed by corpus, grader, parameters and the provider's resolved model snapshot. When a provider moves a -latest alias underneath a stable name, the affected record is invalidated instead of looking fresh. |
Grader validity was established before use: it agreed with the reference grading on 96.5% of a validation sample (Cohen's κ = 0.91, n = 173), and showed no self-preference. Its lowest agreement was on answers from its own model family.
Each model is measured on ~250 graded prompts, executed through the same serving path your requests take, so end-to-end latency is captured at the same time. The catalogue is measured continuously as models are added, so the number of models on the axis grows; every score on it was produced by the same instrument.
Tiers, and why models inside one are not ranked
A tier is a quality floor. Every model the router can select in a tier has been measured to clear that tier's floor, which is what makes it safe for your strategy to optimise for cost or latency inside a tier. Models that clear no tier's floor, or whose measurement was inconclusive, are marked provisional and excluded from automatic routing — they remain in the catalogue, and you can still select them explicitly.
At this sample size the instrument resolves quality differences of about 0.046. Models closer
together than that are treated as tied rather than ranked, because an order between them
would be measurement noise. This is why balanced and cheapest are usually better choices than
highest_quality within a tier: the quality difference they would be trading against is often
not real, whereas the cost and latency differences are.
Two consequences worth knowing:
- Model outputs are not deterministic. Re-asking the same model the same questions, roughly 8% of prompts change between correct and incorrect. Published intervals are wider than that re-measurement spread, so they are conservative rather than blind to it.
- Provisional means unmeasured, not poor. A newly added model is withheld from automatic
routing until it has been through the instrument, and some of the strongest models in the
catalogue are provisional at any given time simply because they arrived recently. If you want
one of them, name it explicitly rather than waiting for
autoto find it.
What automatic routing delivers
Measured against routing every request to a frontier model, the baseline most teams actually start from:
~58% lower cost, retaining ~96% of that model's quality — measured on workload types the router had never seen during training.
The deployed checkpoint is measured on business prose — extraction, summarisation, drafting, classification and analysis. It is trained on that corpus plus a second reasoning-and-knowledge corpus, which improves it without entering any measurement. Benchmark: enterprise workloads is the full account — baselines, per-industry results, and the check that separates choosing well from merely choosing cheaply.
The dial moves that trade-off, and both settings are measured, not interpolated:
Quality is each request's actual graded score from the model that served it, and cost is each request's actual token spend at current provider prices. Neither is a per-tier constant assumed in advance.
The baseline matters, and this is the honest version.
Automatic routing is not cheaper, it costs more. What it buys is quality: 0.801 against a
mid-tier model's 0.632 on the same held-out set. If a cheap model already handles your traffic
acceptably, pinning it is the correct answer and auto has little to offer.
So auto is built for teams over-provisioned to a frontier model, paying frontier prices for
the majority of requests that do not need one.
Routing overhead itself is well under a millisecond, with no additional model call and no additional tokens billed. The decision is a small classifier over features computed directly from the request text; it needs no embedding model, so there is no second inference in the hot path.
How these numbers are produced
Two measurement paths stand behind this page:
- Cost and quality trade-offs (the figures above) come from offline simulation over a large body of prompts whose answers had already been produced and graded across many models independently of us. This makes it possible to evaluate thousands of routing decisions, and every counterfactual baseline, without the result depending on the day it was run.
- Model quality, latency and cost are measured live, through the same serving path your requests take.
A prompt's assignment to the held-out set is a function of the prompt itself, not of its position in the corpus. The held-out set therefore does not move when the dial, the sample size or the corpus changes, so two runs quoted here are comparable, and a better number cannot be produced by reshuffling. Every reported figure is quoted alongside the baselines it should be judged against: the cheapest tier used for everything, the mid tier used for everything, the frontier tier used for everything, and a perfect-knowledge oracle.
Re-measurement
| What | Cadence |
|---|---|
Observed latency per (provider, model) | Recomputed hourly over a rolling 5-day window |
| Model availability | Checked daily; models that stop responding are excluded from routing |
| Catalogue prices and context windows | Audited against live provider data; corrections applied to the catalogue |
| Model quality | Re-measured when a model is added, or when its provider snapshot changes |
Because quality records are keyed by the provider's resolved model snapshot, a provider silently updating a model behind a stable alias invalidates that model's record rather than leaving a stale figure in place.
See also
- Benchmark: enterprise workloads: the router measured on business prose — extraction, summarisation, drafting and analysis — against workload types it had never seen during training
- Benchmark: MMLU-Pro: the same router on a public academic benchmark — where routing has least to offer, and the page says so
- Intelligent routing: how the routing decision is made
- Routing: configuring strategies, fallback and aliases