Benchmark: enterprise workloads
Routing performance reports how the router behaves on a general prompt corpus. MMLU-Pro reports how it behaves on an academic multiple-choice benchmark. This page reports a third thing, and the one closest to how enterprise teams actually use an LLM: business prose — extraction, summarisation, drafting, classification and analysis, written the way people in real roles write.
Headline
58.8% lower cost, retaining 95.3% of that model's quality — measured on workload types the router had never seen during training.
The baseline is a frontier model answering everything. That is the right comparison if the alternative you are weighing is "send it all to the best model", and the wrong one if you would otherwise pick a single cheap model — on this workload a well-chosen cheap model is competitive. What this does not show has the numbers.
Per 100,000 requests, on the held-out set:
| Configuration | Quality | Cost / 100k | vs frontier |
|---|---|---|---|
| Cheap model, everything | 0.622 | $41 | −98.0% |
| Mid-tier model, everything | 0.632 | $29 | −98.6% |
| Automatic routing - balanced | 0.801 | $846 | −58.8% |
| Automatic routing - cost-leaning | 0.759 | $457 | −77.8% |
| Frontier model, everything | 0.840 | $2,056 | — |
Routing also cuts median latency, because most requests are answered by a faster model: ~33% lower on the balanced setting and ~50% lower on the cost-leaning one, against ~25.1s if every request went to the frontier model.
Two settings, one dial
The trade-off is a dial, not a fixed point. Both settings below were measured the same way, on the same held-out combinations:
| Setting | Cost saved | Quality retained | Latency | Traffic to frontier model |
|---|---|---|---|---|
| Balanced (λ = 0.1) | 58.8% | 95.3% | −33% | 62% |
| Cost-leaning (λ = 0.2) | 77.8% | 90.4% | −50% | 42% |
Both figures are from the held-out set — the six workload combinations withheld from training —
scored on recorded answers from the three anchor models, not on a live end-to-end run. Both beat a random router escalating at
the same rate, on cost and on quality, across 300 random reassignments: p < 0.001 for balanced,
p ≤ 0.005 for cost-leaning. Settings between and beyond these are available; the trade-off is
measured at each, not interpolated.
What was measured
A corpus of 2,328 business prompts spanning five industries, eight departments and twelve task types — extraction, summarisation, classification, drafting, structured output, multi-step analysis and others. Each prompt exists in ten writing styles, including terse, verbose, poorly written, multilingual, and buried in irrelevant context, so the corpus reflects how requests actually arrive rather than how they would be written by someone trying to make a router look good.
Every prompt was answered by each tier's model and graded by an independent LLM grader against a rubric written for that task type. Cost is each request's real token spend at provider prices, and quality is the answer's actual graded score. Neither is a per-tier constant assumed in advance.
What the router was trained on
Worth separating from what it was measured on. The training set is 2,161 prompts: 1,159 from this business corpus, plus 1,002 reasoning-and-knowledge prompts from a second corpus (mathematics, multiple choice, retrieval-style questions) that were measured on their own three-rung model ladder.
The second corpus is included because it helps: it buys roughly 4–12% lower cost at matched quality. Cost-aware training can use it even though its ladder is a different set of models, because the label is a position on a ladder — cheapest, middle, dearest — and that means the same thing on both. A label naming specific models could not have been pooled this way.
Every figure on this page is measured on business prose only — 276 validation and 309 held-out prompts, none of them from the second corpus. The mixed training set makes the router better; it does not flatter the measurement.
How the held-out set works
The corpus is organised into workload combinations — an industry, a department, an operation and a task type, for example banking / risk & compliance / transaction review / extraction.
Six entire combinations were withheld from every stage: rubric design, feature selection, model selection and tuning. They were scored once, at the end, with the configuration already fixed. So the headline figure is not "how well does it do on prompts like the ones it learned"; it is how well does it do on kinds of work it has never seen.
This is a deliberately harder test than a random prompt split. Ten writing styles of one scenario are near-duplicates of each other, so splitting prompts at random would place a prompt's own siblings in the training set and report memorisation as skill. Splitting whole combinations prevents that.
Cross-validated results, by industry
The held-out set is six combinations across four industries, which is too thin for a per-industry cut. For that, every one of the 32 workload combinations is held out in turn, so each prompt is still scored by a model that never saw its combination — just with far more of them.
Balanced (λ = 0.1)
| Workload | Prompts | Cost saved | Quality retained |
|---|---|---|---|
| SaaS | 495 | −41.0% | 98.7% |
| Banking | 487 | −59.8% | 96.9% |
| E-commerce | 424 | −52.6% | 93.9% |
| Healthcare | 270 | −48.6% | 93.5% |
| Consulting* | 68 | −86.5% | 76.6% |
| All workloads | 1,744 | −55.1% | 95.3% |
Cost-leaning (λ = 0.2)
| Workload | Prompts | Cost saved | Quality retained |
|---|---|---|---|
| SaaS | 495 | −66.9% | 98.1% |
| Banking | 487 | −84.5% | 93.4% |
| Healthcare | 270 | −82.7% | 89.2% |
| E-commerce | 424 | −77.4% | 90.1% |
| Consulting* | 68 | −95.6% | 73.5% |
| All workloads | 1,744 | −80.2% | 92.5% |
* Consulting is 68 prompts across two workload combinations and should not be read as an industry result. Its figures barely move between settings, which indicates a property of those two combinations rather than of the dial. It is shown for completeness because it is included in the totals.
The cross-validated and held-out figures agree closely — −55.1% against −58.8% on balanced, −80.2% against −77.8% on cost-leaning — which is the point of reporting both: the result does not depend on which combinations happened to be withheld.
Savings and quality do not move together across industries. SaaS gives up the least quality (98.7%) and saves the least (−41.0%); Banking saves half again as much for two points more quality. That is the corpus telling you something real: workloads differ in how often a cheaper model is genuinely sufficient, and a single blended figure hides it.
Is the saving real, or just spending less?
Any router that sends less traffic to the frontier model will look cheaper. The question is whether it is choosing well or merely choosing cheaply.
To separate those, the router's decisions are compared against a random router that escalates exactly as often — the same number of requests to each tier, assigned to different prompts. If the saving came from escalation rate alone, the two would score the same.
They do not. On the held-out set the router delivers both lower cost and higher quality than the
random router at the same escalation rate, across 300 random reassignments — p < 0.001 on both at
the balanced setting, and p ≤ 0.005 at the cost-leaning one. The saving comes from which requests
are escalated, not how many.
How the router decides
The router is trained directly on the decision it has to make, rather than on a proxy for it.
For every prompt in the corpus, each tier's actual quality and actual cost are known. The training
target is the tier that maximises quality − λ × cost — the cheapest model that is genuinely good
enough for that request. Each example is weighted by how much the decision matters: a prompt where
every tier performs equally well carries almost no weight, and a prompt where the wrong choice is
expensive carries a lot.
λ is the dial. A lower λ buys quality, a higher λ buys savings — the two settings in the headline table are λ = 0.1 and λ = 0.2. Each setting is its own measured operating point rather than a curve interpolated from one run.
The decision costs no additional model call and no additional tokens. It is a small classifier over features computed directly from the request text — length and structure, readability, constraint, output-format demands, script and language signals — evaluated in well under a millisecond.
What this measures, and what it leaves out
Routing happens in two independent stages, and this page measures only the first.
| Stage | Decides | Controlled by |
|---|---|---|
| Tier selection | which quality tier a request needs | the λ dial — the two settings on this page |
| Model selection | which model within that tier serves it | your routing strategy: balanced, cheapest, fastest, highest-quality |
Every figure here serves each tier with its anchor model — the model that defines that tier's quality floor. A tier normally holds many models, and the configured strategy ranks them by cost, latency and quality. That second stage is switched off in this measurement, for a simple reason: only the three anchor models were run across all 2,328 prompts, so there is no second model in a tier to choose between.
A consequence worth being clear about:
- These are not the
cost-first/balanced/quality-firstarms reported on the MMLU-Pro page. Those vary the model-selection strategy over a multi-model pool. The settings here vary the tier decision. The two are orthogonal and compose: λ chooses the tier, your strategy chooses within it.
See also
- Routing performance: how model quality, cost and latency are measured
- Benchmark: MMLU-Pro: the router on a public academic benchmark
- Intelligent routing: how a request reaches a model