Model routing and cost

Routing is static. Prompt caching drives the bill, and the model decides the outcome. An agent run gets one model from its data class (GLM-5.3 for public, Claude Opus 5.5 for internal), whatever its tier. The tier sets the run’s caps and its team, not its model. Nothing classifies a task yet: every task is triaged standard by the static fallback, and 10% go to frontier as a control group. Typically more than 90% of the input tokens agents send are cache reads, which cuts the bill about fourfold. On harder tasks the model changed the outcome more than any routing could: Claude was accepted on 13 of 15 issues, GLM-5.3 on none, in either harness.

This page owns the routing, tuning and cost angle. How the gateways authenticate and what they carry is on Gateways. What runs where is on Status, which wins over this page if the two disagree.

Dollar figures on this page are examples from the reference host, at the list prices its meter.prices pins.

What is built

Every routing decision, end to end

There are two callers and seven decisions. Only two of the seven look at the prompt: the Semantic Router on MoM, and the planned triage classifier (once per task, never per request).

    flowchart TB
  subgraph human["Humans and coding clients: ai-gateway (Envoy AI Gateway)"]
    C["Client: OpenCode, Continue, OpenWebUI, promptfoo"] -->|"model: name or MoM"| SR{"1 · Semantic Router<br/>only when model is MoM or auto"}
    SR -->|"rewrites body.model"| XM["x-ai-eg-model header"]
    C -->|"explicit name, no classification"| XM
    XM --> AR{"2 · AIGatewayRoute<br/>header match"}
    AR -->|"tier-frontier"| ZAI1["Z.ai glm-5.3"]
    AR -->|"xplane-qwen-coder"| CAN{"3 · LoRA canary<br/>90 / 10 weights"}
    AR -->|"other xplane-*"| V["vLLM Service"]
    CAN --> V
  end
  subgraph agent["Agent runs: agentgateway agent-router"]
    I["Issue labelled factory/ready"] --> T{"4 · Triage, once per task<br/>tier and template"}
    T --> R{"5 · Role<br/>reviewer takes the other tier"}
    R -->|"tiers.tier.model or reviewerModel"| N["Logical name<br/>agent-default, tier-*"]
    N --> D{"6 · Data class<br/>audience picks the listener"}
    D -->|"public :8080"| AL1{"7 · modelAliases"}
    D -->|"internal :8081"| AL2{"7 · modelAliases"}
    AL1 --> ZAI2["Z.ai glm-5.3, flashx, flash"]
    AL2 --> ANT["Anthropic opus-5-5, sonnet-5-5, haiku-5-5"]
  end
  

Decisions 4–7 are pinned by design: every name maps to one model at weight 100, with no fallback and no canary, so nothing re-routes a run mid-trajectory. Escalation is a new run at another tier, never a gateway failover (routing design).

Serving side: ai-gateway

The MoM cascade as configured

Semantic Router 0.2.0 (flux/sources/ocirepo-vllm-semantic-router.yaml) still uses the v0.2 schema. A ModernBERT domain classifier maps the prompt to an MMLU category, keyword and length rules add evidence, and the highest-priority decision that matches wins:

    flowchart TD
  P["model: MoM"] --> J{"prompt_guard<br/>jailbreak above 0.7"}
  J -->|"yes"| B["blocked"]
  J -->|"no"| CD{"domain code<br/>or code_keywords"}
  CD -->|"yes"| RK{"reasoning_keywords<br/>or prompt over 500 tokens"}
  RK -->|"yes · 110"| QCR["xplane-qwen-coder<br/>use_reasoning: true"]
  RK -->|"no · 100"| QC["xplane-qwen-coder"]
  CD -->|"no"| MP{"domain math or physics"}
  MP -->|"yes · 90"| Q8R["xplane-qwen3-8b<br/>thinking on"]
  MP -->|"no"| ML{"domain multilingual"}
  ML -->|"yes · 80"| Q8["xplane-qwen3-8b"]
  ML -->|"no · 50 or no match"| Q8D["xplane-qwen3-8b<br/>default_model"]
  

Two behaviours to know before you rely on MoM.

• It sits in front of every request, and fails closed. The filter is on the http listener, not only on MoM, and has no failure_mode_allow, with a 60 s message_timeout (infrastructure/base/envoy-ai-gateway/envoypatchpolicy-semantic-router.yaml). An unreachable router fails every request on that listener, tier-frontier included. A router that is up but not rewriting lets MoM reach vLLM, which answers 404 for an unknown model (the HelmRelease’s own note). The designed fix, not built: two replicas, a PDB, and fail-open with a 2 s timeout.
• Classification costs about 250–300 ms (gateways). A client that knows its model names it, and the router passes it through untouched. FIM traffic never uses MoM.

The designed cascade (not built)

The routing design replaces fleet names with roles that claims opt into (spec.gateway.aliases), and adds the one escalation that leaves the cluster:

Its verdict rule (SC-9) is set in advance: unless routed MoM reaches 95% of always-frontier’s promptfoo pass rate at 50% of its cost or less over 14 nightly runs, MoM defaults to mom-standard. It is the right bar: an independent evaluation found router gains “track selected-tier composition more closely than demonstrated task-specific targeting” (arXiv 2608.14641). A router has to beat a fixed tier, not simply exist.

After the model is chosen: canary, endpoint picker, autoscaling

A routing change on the serving side is a capacity change too. Moving a category from xplane-qwen3-8b to xplane-qwen-coder moves its share of the batch-saturation trigger, and the LoRA canary shares the coder’s batch: adapter traffic scales the base model.

Agent side: tiers, data classes, names

How a run gets its model

    flowchart TB
  subgraph tier["1 · Task received: the tier"]
    direction LR
    Ctl(["controlPercent: 10<br/>this task in the<br/>control group?"])
    Ctl -->|"yes"| Fr["tier frontier"]
    Ctl -->|"no"| Cls(["classifier<br/>answered?"])
    Cls -->|"no: unbuilt<br/>today"| St["tier standard<br/>fallback: static"]
    Cls -->|"yes: design"| Tier["its tier, nudged<br/>up when unsure"]
  end
  subgraph tpl["2 · The template, from class and tier"]
    direction LR
    Tpl(["template from<br/>class and tier"])
    Tpl -->|"docs-links<br/>below frontier"| Solo["solo"]
    Tpl -->|"other class below<br/>frontier, or docs-links<br/>at frontier"| Pair["pair"]
    Tpl -->|"other class<br/>at frontier"| Trio["trio"]
    Pair --> Rev["reviewer: the other tier<br/>standard after frontier,<br/>frontier otherwise"]
    Trio --> Rev
  end
  subgraph model["3 · The model and provider"]
    direction LR
    Name["model name:<br/>tiers.tier.model or<br/>templates.name.reviewerModel"] --> DC(["spec.dataClass"])
    DC -->|"public"| G["GLM via Z.ai"]
    DC -->|"internal"| A["Claude via the<br/>Anthropic API"]
  end
  tier -->|"the tier"| tpl
  tpl -->|"solo, or the reviewer's tier"| model
  

The aliases, identical on both Z.ai backends and mirrored on Anthropic’s (infrastructure/base/agent-gateway/llm.yaml):

Any other name is refused 403 by the profile’s model rule (policy-capabilities.yaml) before an upstream call; the transformation’s 400 stays behind it. A run can still ask for any of the four names in its class: the budgets bound what that costs.

Why every tier still asks for agent-default

# tooling/base/agent-factory/helm-values-configmap.yaml
tiers:
  light:    {model: agent-default, runTokens: 300000,  taskTokens: 600000,  runMinutes: 20, runUSD: 0.51, taskUSD: 1.02}
  standard: {model: agent-default, runTokens: 1500000, taskTokens: 3000000, runMinutes: 45, runUSD: 2.55, taskUSD: 5.10}
  frontier: {model: agent-default, runTokens: 4000000, taskTokens: 8000000, runMinutes: 90, runUSD: 6.80, taskUSD: 13.60}

The gateway serves the tier-* names on both listeners, so moving a tier to its own model is a one-line change (see tuning). Nobody has measured whether glm-5.3-flashx or glm-5.3-flash can do a factory task, so the change has not been made. Until it is:

• the tier sets caps, minutes and the team, not the model. The control group measures budget fit, not model fit;
• the reviewer runs the implementer’s model. “The other tier” changes its caps only. ADR-0058 keeps it that way on public work, so no public data goes to Anthropic for review (ADR-0058).

Complexity routing for tasks: kept out of the request path

Source: factory design and routing design §3. None of it is built: every task reads classifier: static, fallback: static.

What a call costs

One price table

meter.prices, in tooling/base/agent-factory/helm-values-configmap.yaml, is the only place a price is edited. Every dollar figure on this page comes from it, and task check fails when a copy drifts:

A price change is the row’s edit, python3 scripts/ci/gen-prices.py --write (it renders the gateway’s agent-model-catalog overlay too) and the cost string the CI gateway check prints for infrastructure/base/agent-gateway/policy-budgets.yaml, in one commit.

meter.prices, USD per million tokens. Example (the reference host’s list prices):

Haiku is at its over-100K-prompt rate, which an agent’s context soon reaches. Over-counting is the safe direction for a cap.

A nightly workflow, .github/workflows/price-drift.yaml, compares every row with genai-prices, LiteLLM and models.dev. A disagreement, or a row a catalog stops listing, opens or updates one issue labelled prices-drift, and the first clean run closes it. The check never edits the table: catalogs copy the providers’ page mistakes, so a person confirms each finding on the provider’s page first.

The charge series

The gateways count cache tokens inside input (see the meter’s formula below), so the recording rule holds a charge, not a price: what one counted token of each type adds to the cost.

A cost over a window is one join, to which each reader adds its own selector:

sum(
  sum by (gen_ai_request_model, gen_ai_token_type) (
    increase(gen_ai_client_token_usage_sum{gen_ai_token_type=~"input|output|cached_input|cache_creation_input"}[24h])
  )
  * on (gen_ai_request_model, gen_ai_token_type) group_left() llm_gateway:charge_usd_per_mtoken
) / 1e6

The reply the wire test sends on Opus, 1,050 input of which 900 read and 50 written, and 10 output, reads 1050 × 4.00 − 900 × 3.80 + 50 × 1.00 + 10 × 20.00 = 1,030 per million, $0.00103: the meter’s 100 × 4.00 + 900 × 0.20 + 50 × 5.00 + 10 × 20.00. The list-price rule it replaced read 4.3× that. reasoning is inside output and has no charge. The join prices the whole window at the current charges; what each run was charged stays in maku_factory_run_cost_usd and the Task’s costMicroUSD.

The meter’s formula

    flowchart TB
  subgraph meter["Metering"]
    direction LR
    GW["agentgateway<br/>agentgateway_gen_ai_<br/>client_token_usage"] -->|"VMPodScrape relabel<br/>input_cache_read<br/>to cached_input"| VM[("gen_ai_client_token_usage_sum<br/>by ar_agent, gen_ai_request_model,<br/>gen_ai_token_type")]
    VM -->|"meter.query,<br/>every 30 s"| M["run meter"]
    P["meter.prices<br/>keyed by served model"] --> M
  end
  subgraph cost["Cost and caps"]
    direction LR
    F["fresh = input - cached_input<br/>- cache_creation_input<br/>cost = fresh x input<br/>+ cached x cacheRead<br/>+ creation x cacheWrite<br/>+ output x output"]
    F --> C1{"run cost at or<br/>over runUSD?"}
    C1 -->|"yes"| Rev["revoke: budget-run"]
    C1 -->|"no"| C2{"task cap still holds<br/>one whole run?"}
    C2 -->|"no"| Esc["escalate: budget-task"]
    F --> Met["maku_factory_run_cost_usd<br/>maku_factory_model_cost_usd_total<br/>Task status.runs.costMicroUSD"]
  end
  meter -->|"the run meter computes"| cost
  

• Keyed by the served model, not the logical name: agent-default is GLM-5.3 on one listener and Opus on the other. A model with no price row is never treated as free: its run falls back to its token caps and AgentModelUnpriced fires.
• The input series includes cache tokens on every path this platform runs: Z.ai’s OpenAI usage, and agentgateway’s Chat Completions to Messages translation, streamed or not. test-agent-gateway-cache-wire asserts the plain and the streamed reply at every agentgateway pin; the meter’s comment that a stream leaves them out predates the measurement. The meter subtracts the cache tokens unless input is smaller than them, in which case it prices all of input as fresh (inputExcludesCache, Maku internal/factory/meter/meter.go) and counts the run in maku_factory_meter_input_excludes_cache_runs_total. AgentMeterInputExcludesCache fires on it: the charge series would under-count such a run, down to a negative cost.
• Overshoot is one scrape plus one response. The run is revoked on the meter’s next tick, and calls already in flight complete.

Worked example: a real run

Example (the reference host): an implementer on a standard task, public, served by glm-5.3, which opened a pull request. Its gateway counters:

The meter recorded costMicroUSD: 1648658 on the Task: an exact match. The same run under the old token cap would have been revoked at 1.5 M of its 4.48 M tokens, with the work unfinished. Priced with no cache discount it would show as $6.39.

The same token mix on other models, as arithmetic from meter.prices and not a measurement:

With caching, Opus is not ten times GLM. Opus’s cache read ($0.20) is cheaper than GLM-5.3’s ($0.26), so on a 94%-cached run the list-price gap narrows from about 3× on input to 1.7× overall. Anthropic does bill cache writes ($5.00 per million on Opus), which this estimate leaves out. One measured internal run, a triager on Opus 5.5, cost $0.477 for 303,477 input tokens, 85% of them cache reads.

Budgets: five layers, one exact

    flowchart TB
  subgraph factory["Factory: exact, in USD at meter.prices"]
    R["run cap: runUSD<br/>0.51 · 2.55 · 6.80<br/>enforced"] --> TK["task cap: taskUSD<br/>1.02 · 5.10 · 13.60<br/>enforced"]
    TK --> D["daily: factoryDailyUSD 42.50, humanDailyUSD 8.50<br/>shadow while enforcePrincipal is false"]
  end
  subgraph gateway["agentgateway: backstop, enforced"]
    B1["B1 per run: 5M ref tokens, $8.50"]
    B2["B2 fleet: 41M ref tokens, $69.70"]
    B6["B6 Anthropic: 10M ref tokens, $17"]
  end
  L["local: 40M raw tokens per hour per sub<br/>always enforced"]
  factory --> gateway --> L
  

Efficiency

Prompt caching is the largest lever

Agent runs typically read 85–96% of their input from cache: GLM-5.3 factory runs 91–96% (cohort results), an internal Opus run 85%. On a whole fleet day that cuts the bill by about 75% against the same tokens with no cache discount.

Caching is also why per-request model switching is a bad idea for agents: a new model starts with a cold cache, and on a 95%-cached run that multiplies the next call’s input cost by about 5.

Caps in USD, not tokens

The hard-task cohort found that the token cap, not the work, decided 11 of the factory’s 15 issues: it counted every cache read as a full token, and runs stopped at about 1.5M tokens and $0.58–0.76. Caps are in USD because a dollar cap prices cache reads at their real rate, so a cache-heavy run gets the work its budget pays for. Example (the reference host), two standard runs ($2.55 cap):

Cheaper tiers: priced, not measured

Moving light and standard to their own names would cut the price of the same tokens to 0.11× and 0.28× (worked example above). Whether those models finish factory tasks is unmeasured. The one model comparison on record points the other way: on harder tasks the gap followed the model.

C ran B’s harness on A’s model and hit the 60-turn cap on all 15 issues, at $1.13–2.33 each. A cheaper model that fails costs more per accepted outcome than a dearer one that succeeds. The cohort’s caveat holds: fifteen issues separate only large differences, and A was stopped by its old token cap.

Self-hosted against an API

The scale-to-zero deadlock is why the floor is fixed: a model with no pod emits no metric to wake it (autoscaling).

Tuning

Every knob, what it trades and how to see the effect. Factory values live in tooling/base/agent-factory/helm-values-configmap.yaml, a gate path a maintainer changes. A meter.prices change must be mirrored in the CEL cost of infrastructure/base/agent-gateway/policy-budgets.yaml: the CI gateway check fails any difference.

Move a tier to its own model:

# tooling/base/agent-factory/helm-values-configmap.yaml
tiers:
  # tier-light is glm-5.3-flash on public, claude-haiku-5-5 on internal (llm.yaml, anthropic.yaml);
  # both are priced in meter.prices, so the USD caps keep working.
  light: {model: tier-light, runTokens: 300000, taskTokens: 600000, runMinutes: 20, runUSD: 0.51, taskUSD: 1.02}

Do it with a measurement in place: run a cohort on the tier with fixed-tier arms, and compare accepted outcomes and USD per accepted outcome.

Change a provider model behind a name (both Z.ai backends, and the gate’s price row):

# infrastructure/base/agent-gateway/llm.yaml, on zai and zai-anthropic
modelAliases:
  tier-standard: glm-5.3-flashx   # a new model here needs its meter.prices row, or the CI gateway check fails

In-flight runs see a mapping change on their next request. Use the factory’s pause first when that matters.

Tune the chat cascade (Semantic Router 0.2 schema):

# infrastructure/base/vllm-semantic-router/helmrelease.yaml
context_rules:
  - name: long_query
    min_tokens: 500    # raise to turn reasoning on for fewer code prompts
    max_tokens: 32000

The complexity signal and role targets need SR 0.3.0. Do not add a complexity rule to the 0.2 config with the documented 0.70–0.75 threshold: it never fires.

Measuring

Signal definitions and alerts are on Observability.

Known gaps