AI inference control plane · router v1

Stop routing by model.Start routing by task.

AICP classifies every request, routes it to the right model, and enforces policy before it executes.

Free for your first month Read a live trace

46%

cheaper than Sonnet on MMLU-Pro

95.7%

of Sonnet's accuracy

<1ms

median routing overhead

measured, not projected · MMLU-Pro, a public benchmark · 1,219 questions

req_8f3a2c · live routing decision

12ms

POST /v1/chat · tier 0 · balanced strategycost 0.40 · latency 0.20 · quality 0.40

Scoring 7 in-tier candidatesdone

Qwen3-30B-A3B-Instruct-2507 selected

Nebius · confidence 86%$0.00042

gpt-oss-120b67

score 67/100$0.00030 est.

gemma-3-27b-it58

score 58/100$0.00015 est.

gemini-2.5-flash55

score 55/100$0.00094 est.

claude-haiku-4.545

score 45/100$0.0023 est.

outcome captured for the Learning Enginerequest succeeded · 2,765 tokens

ML routing that earns its overhead.

The routing pipeline runs in under a millisecond and typically saves 40-80% on inference cost by routing simple requests to smaller, cheaper models, without sacrificing the quality your application needs.

Routing strategies

POST org_routing_configs · payloadcheapest

{
  "strategy": "cheapest",
  "params": {
    "default_prompt_tokens":  "2000",
    "default_output_tokens":  "512"
  }
}

Under the hood

ML complexity classifier

Every request is scored by a purpose-built ML router to predict a small, medium, or large complexity tier. It reads hand-crafted features straight from the request text — no embedding model, no second inference before the real one. Falls back to your configured strategy if the router is ever unavailable.

Observed-latency feedback loop

Real p50 and p95 latency per provider/model is tracked and refreshed hourly, overriding static catalog estimates. Routes naturally shift away from degraded providers without any manual intervention.

Explainable decisions in Request Explorer

Every routing decision is logged with its full reasoning: tier predicted, models scored, policy applied, winner selected. Fully queryable and replayable from the dashboard.

Context compaction

When a prompt exceeds the largest available model's context window, AICP automatically compacts it before routing. Available on Business plans with the VS Code Copilot client.

Capability filtering runs before every strategy

Before any strategy executes, AICP filters the candidate pool to models that support the request's detected requirements: streaming, vision, function-calling, JSON mode, long-context, audio, or reasoning. No config knob.

Eight native adapters. Every OpenAI-compatible endpoint.

Provider adapters are maintained in the AICP core, not wrappers around the provider's SDK. That means consistent retry semantics, streaming behaviour, and capability detection across every provider you connect.

OpenAI

GPT-4o, o1, o3-mini, …

Anthropic

Claude 3.5 Sonnet, Haiku, …

Google Gemini

Gemini 1.5 Pro/Flash, …

Mistral

Medium, Small, Codestral, …

Azure OpenAI

GPT-4o via Azure endpoint

Azure AI Foundry

Phi-3, Llama, Cohere via Hub

AWS Bedrock

Titan, Llama, Claude on AWS

AWS SageMaker

Custom & fine-tuned endpoints

Any OpenAI-compatible endpoint: Nebius, Together, Groq, Fireworks, Ollama, and more

If it speaks the OpenAI chat completions API, AICP can route to it. No custom adapter required. Point it at your base URL and it's available in the routing pool immediately.

base_url: https://…

BYOK, AES-256-GCM encrypted. Keys never leave the control plane in plaintext.

No code? Use the n8n connector instead.

Set policy once. Enforced on every request, across every team.

Engineering leadership shouldn't have to audit each team's model choices manually. Define data residency rules, spend limits, and model allowlists at the org level, and AICP enforces them automatically.

PII detection and redaction (Presidio)

Outbound prompts are scanned for personally identifiable information using Microsoft Presidio. Configurable fail-open or fail-closed. Available on every plan.

Data residency rules

Bind projects or orgs to specific geographic regions. Requests that cannot be satisfied by a compliant model are rejected, never silently routed to a non-compliant provider.

Model allow/blocklists and org-level policies

Define which models each team or project can access. Set org-wide defaults. Override at the project level. Policies are evaluated on every request, not at deploy time.

Budget guards per project

Set monthly spend caps per project. Requests that would breach the cap are rejected with a structured error before a provider is called. No surprise bills.

GDPR-minded by design

PII redaction, data residency, no training on your data

EU AI Act-aware

Architecture supports per-jurisdiction model filtering

ISO 27001

Certification in progress

Every request logged. Every decision replayable.

A ClickHouse-backed logging pipeline captures every prompt, completion, routing decision, latency, and cost across four dashboard views.

Live request volume, cost by model, provider health, and active alerts at a glance.

Requests (30d)

284,425

Total cost

$1,134.20

Avg p50 latency

1.4s

Avg quality score

0.91

eval-service

Response quality scoring and cross-model benchmarking. Run regression comparisons when you add or rotate a model, and catch quality drops before they reach production.

logging-service

Durable ClickHouse request/response trail. Every prompt, completion, latency, cost, and routing decision is stored and queryable. Retention is configurable per plan.

notification-service

Email alerts and outbound webhooks for budget breaches, API key expiry, and provider health incidents. Integrate into your existing on-call tooling.

Start free. Pay only for what you route.

Cloud is cost + 3.5% on your AI spend, with free calls in your first month. Need to self-host? Enterprise is a custom, sales-assisted deployment on your own infrastructure.

Hard-coding AI models is the new technical debt.

Stop writing routing logic. Stop overpaying for inference. Stop guessing which model fits the task. AICP handles all of it, automatically, on every request.

Start free Book a call Read the Docs