Benchmark: MMLU-Pro
Routing performance reports how the router behaves on our own measurement corpus, and enterprise workloads reports how it behaves on business prose. This page reports something different and, for most readers, complementary: how it performs on a public benchmark.
Headline
0.853 accuracy at $5.10 per 1,000 requests — 4.3% less accurate for 46% less cost. Against gpt-oss-120b used for everything, routing buys +4.2 points of accuracy.
MMLU-Pro is where routing has least to offer. It is academic multiple-choice, where one capable mid-tier model does well across every subject, so there is little for a per-request decision to exploit. It is published because it is a fixed, public, mechanically-checkable yardstick — not because it flatters the router. For the workload the router is built for, see enterprise workloads.
The full table
All configurations answered the same 1,219 questions, over the same pool of models, with all three tiers populated as the catalogue has them.
Routing has two independent controls, and both are shown separately rather than blended into one "setting":
- λ — the tier dial, fixed when the router is trained. Lower λ escalates more.
- Routing policy — how a model is chosen within the tier λ selected. Shown for cost-only, balanced (50 cost / 50 quality / 0 latency) and quality-only.
| λ | Policy | Accuracy | Cost / 1k | → tier 2 |
|---|---|---|---|---|
| 0.20 | cost | 0.842 | $4.26 | 37% |
| 0.20 | balanced | 0.842 | $4.26 | 37% |
| 0.20 | quality | 0.842 | $4.31 | 37% |
| 0.10 | cost | 0.853 | $5.10 | 46% |
| 0.10 | balanced | 0.853 | $5.10 | 46% |
| 0.10 | quality | 0.853 | $5.11 | 46% |
| 0.05 | cost | 0.852 | $5.52 | 51% |
| 0.05 | balanced | 0.852 | $5.52 | 51% |
| 0.05 | quality | 0.852 | $5.53 | 51% |
| Single model, everything | Accuracy | Cost / 1k |
|---|---|---|
| Nemotron-3-Nano | 0.791 | $0.75 |
| Claude Haiku 4.5 | 0.797 | $3.50 |
| gpt-oss-120b | 0.811 | $0.70 |
| Claude Sonnet 4.6 | 0.891 | $9.49 |
Accuracy is the share of questions answered correctly. Cost is actual measured token spend at current provider prices — not an estimate, not a per-tier constant assumed in advance.
λ moves the result; the policy does not — accuracy spans 0.842 to 0.853 across λ, while the three policies differ by at most 0.05%. That is a fact about the pool of models that have answered these specific questions.
Why you can trust these numbers
Benchmark results are easy to produce and easy to get wrong. Four things were done specifically to make this one hold up.
1. The router was not tested on what it was trained on
This is the failure that quietly invalidates most published routing benchmarks.
The router's training set includes a reasoning-and-knowledge corpus that draws on MMLU-Pro. Exactly one question in the evaluation set turned out to appear in it, and that question was removed. The number is small because the current router is trained mostly on business prose rather than on academic benchmarks — but the check is run regardless of how small the answer is expected to be, because "we assumed the overlap was negligible" is how contaminated benchmarks get published.
2. Comparisons are paired, on identical questions
Every configuration answered the same 1,219 questions, so differences are computed per question rather than between independent averages. The question count is set by coverage: every model in the pool must have answered a question for it to be included, and the tier-2 model is the scarcest.
3. Grading is mechanical
MMLU-Pro answers are option letters, so correctness is decided by exact match. No model grades another model. An answer that never states a choice counts as wrong, not as a missing sample — so a model cannot improve its score by failing to follow instructions.
4. The harness is calibrated against an independent source
The strongest check available: run models a third party has already published figures for, and see whether we reproduce them.
| Model | Our harness | Artificial Analysis | Difference |
|---|---|---|---|
| gpt-oss-120b | 0.784 | 0.808 | −2.4 pt |
| Claude Haiku 4.5 | 0.779 | 0.760 | +1.9 pt |
Our measurements land within ±2.5 points of an independent evaluator, in both directions, so there is no systematic bias inflating them. Two reference points is a small sample, but it is two more than most published routing comparisons offer.
What this does not show
The same discipline applies to the limits, and on this benchmark they are substantial.
MMLU-Pro is narrow. Academic multiple-choice across 14 subjects. Routing has least to offer where every tier performs about equally, and a benchmark where one capable mid-tier model does well across all subjects is exactly that kind of workload. Traffic with genuinely mixed difficulty is where prompt-level routing has room — and it is where you should measure before relying on a number.
These figures replay recorded results. Each model's answers to these questions were produced and graded through the live serving path; the routing comparison then replays them, so every configuration is scored on identical answers rather than on separate live runs made at different moments. This removes run-to-run variation but means the numbers are not a fresh end-to-end execution.
See also
- Benchmark: enterprise workloads: the same router on business prose, where it does beat its random control
- Routing performance: how model quality, latency and cost are established
- Intelligent routing: how a request reaches a model