Model routing and cost

Today, routing is static. Prompt caching drives the bill, and the model decides the outcome. An agent run gets one model from its data class (GLM-5.3 for public, Claude Opus 5.5 for internal), whatever its tier. The tier sets the run’s caps and its team, not its model. Nothing classifies a task yet: every task is triaged standard by the static fallback, and 10% go to frontier as a control group. On gcp-0, 95% of the input tokens agents sent in the last 24 hours were cache reads, which cut the bill about fourfold. On harder tasks the model changed the outcome more than any routing could: Claude was accepted on 13 of 15 issues, GLM-5.3 on none, in either harness.

This page owns the routing, tuning and cost angle. How the gateways authenticate and what they carry is on Gateways. What runs where is on Status, which wins over this page if the two disagree.

What is real today

Every routing decision, end to end

There are two callers and seven decisions. Only two of the seven look at the prompt: the Semantic Router on MoM, and the planned triage classifier (once per task, never per request).

    flowchart TB
  subgraph human["Humans and coding clients: ai-gateway (Envoy AI Gateway)"]
    C["Client: OpenCode, Continue, OpenWebUI, promptfoo"] -->|"model: name or MoM"| SR{"1 · Semantic Router<br/>only when model is MoM or auto"}
    SR -->|"rewrites body.model"| XM["x-ai-eg-model header"]
    C -->|"explicit name, no classification"| XM
    XM --> AR{"2 · AIGatewayRoute<br/>header match"}
    AR -->|"tier-frontier"| ZAI1["Z.ai glm-5.3"]
    AR -->|"xplane-qwen-coder"| CAN{"3 · LoRA canary<br/>90 / 10 weights"}
    AR -->|"other xplane-*"| V["vLLM Service"]
    CAN --> V
  end
  subgraph agent["Agent runs: agentgateway agent-router"]
    I["Issue labelled factory/ready"] --> T{"4 · Triage, once per task<br/>tier and template"}
    T --> R{"5 · Role<br/>reviewer takes the other tier"}
    R -->|"tiers.tier.model or reviewerModel"| N["Logical name<br/>agent-default, tier-*"]
    N --> D{"6 · Data class<br/>audience picks the listener"}
    D -->|"public :8080"| AL1{"7 · modelAliases"}
    D -->|"internal :8081"| AL2{"7 · modelAliases"}
    AL1 --> ZAI2["Z.ai glm-5.3, flashx, flash"]
    AL2 --> ANT["Anthropic opus-5-5, sonnet-5-5, haiku-5-5"]
  end
  

Decisions 4–7 are pinned by design: every name maps to one model at weight 100, with no fallback and no canary, so nothing re-routes a run mid-trajectory. Escalation is a new run at another tier, never a gateway failover (SP4 S7, S8).

Serving side: ai-gateway

The MoM cascade as configured

Semantic Router 0.2.0 (flux/sources/ocirepo-vllm-semantic-router.yaml) still uses the v0.2 schema. A ModernBERT domain classifier maps the prompt to an MMLU category, keyword and length rules add evidence, and the highest-priority decision that matches wins:

    flowchart TD
  P["model: MoM"] --> J{"prompt_guard<br/>jailbreak above 0.7"}
  J -->|"yes"| B["blocked"]
  J -->|"no"| CD{"domain code<br/>or code_keywords"}
  CD -->|"yes"| RK{"reasoning_keywords<br/>or prompt over 500 tokens"}
  RK -->|"yes · 110"| QCR["xplane-qwen-coder<br/>use_reasoning: true"]
  RK -->|"no · 100"| QC["xplane-qwen-coder"]
  CD -->|"no"| MP{"domain math or physics"}
  MP -->|"yes · 90"| Q8R["xplane-qwen3-8b<br/>thinking on"]
  MP -->|"no"| ML{"domain multilingual"}
  ML -->|"yes · 80"| Q8["xplane-qwen3-8b"]
  ML -->|"no · 50 or no match"| Q8D["xplane-qwen3-8b<br/>default_model"]
  

Two behaviours to know before you rely on MoM.

• It sits in front of every request, and fails closed. The filter is on the http listener, not only on MoM, and has no failure_mode_allow, with a 60 s message_timeout (infrastructure/base/envoy-ai-gateway/envoypatchpolicy-semantic-router.yaml). An unreachable router fails every request on that listener, tier-frontier included (SP4 R13). A router that is up but not rewriting lets MoM reach vLLM, which answers 404 for an unknown model (the HelmRelease’s own note). The fix is SP4 PR 4, not built: two replicas, a PDB, and fail-open with a 2 s timeout.
• Classification costs about 250–300 ms (gateways). A client that knows its model names it, and the router passes it through untouched. FIM traffic never uses MoM.

The designed cascade (SP4, not built)

The SP4 design replaces fleet names with roles that claims opt into (spec.gateway.aliases), and adds the one escalation that leaves the cluster:

Its verdict rule (SC-9) is set in advance: unless routed MoM reaches 95% of always-frontier’s promptfoo pass rate at 50% of its cost or less over 14 nightly runs, MoM defaults to mom-standard. It is the right bar: an independent evaluation found router gains “track selected-tier composition more closely than demonstrated task-specific targeting” (arXiv 2608.14641). A router has to beat a fixed tier, not simply exist.

After the model is chosen: canary, endpoint picker, autoscaling

A routing change on the serving side is a capacity change too. Moving a category from xplane-qwen3-8b to xplane-qwen-coder moves its share of the batch-saturation trigger, and the LoRA canary shares the coder’s batch: adapter traffic scales the base model.

Agent side: tiers, data classes, names

How a run gets its model

    flowchart TD
  Iss["Task received"] --> Ctl{"controlPercent: 10<br/>this task in the control group?"}
  Ctl -->|"yes"| Fr["tier frontier"]
  Ctl -->|"no"| Cls{"classifier answered?"}
  Cls -->|"no: unbuilt today"| St["tier standard<br/>fallback: static"]
  Cls -->|"yes: design"| Tier["its tier, nudged up when unsure"]
  Fr --> Tpl{"template from class and tier"}
  St --> Tpl
  Tier --> Tpl
  Tpl -->|"docs-links below frontier"| Solo["solo"]
  Tpl -->|"other class below frontier, or docs-links at frontier"| Pair["pair"]
  Tpl -->|"other class at frontier"| Trio["trio"]
  Pair --> Rev["reviewer: the other tier<br/>standard after frontier, frontier otherwise"]
  Trio --> Rev
  Rev --> Name["model name: tiers.tier.model<br/>or templates.name.reviewerModel"]
  Solo --> Name
  Name --> DC{"spec.dataClass"}
  DC -->|"public"| G["GLM via Z.ai"]
  DC -->|"internal"| A["Claude via the Anthropic API"]
  

The aliases, identical on both Z.ai backends and mirrored on Anthropic’s (infrastructure/base/agent-gateway/llm.yaml):

Any other name is set to null by a CEL transformation and refused 400 before an upstream call (gate AG12). A run can still ask for any of the four names in its class (SP4 R10): the budgets bound what that costs.

Why every tier still asks for agent-default

# tooling/base/agent-factory/helm-values-configmap.yaml
tiers:
  light:    {model: agent-default, runTokens: 300000,  taskTokens: 600000,  runMinutes: 20, runUSD: 0.51, taskUSD: 1.02}
  standard: {model: agent-default, runTokens: 1500000, taskTokens: 3000000, runMinutes: 45, runUSD: 2.55, taskUSD: 5.10}
  frontier: {model: agent-default, runTokens: 4000000, taskTokens: 8000000, runMinutes: 90, runUSD: 6.80, taskUSD: 13.60}

When the factory was built the tier-* names answered 404 (plan R11). The gateway has served them on both listeners since AGW-8, so moving a tier to its own model is now a one-line change (see tuning). Nobody has measured whether glm-5.3-flashx or glm-5.3-flash can do a factory task, so the change has not been made. Until it is:

• the tier sets caps, minutes and the team, not the model. The control group measures budget fit, not model fit;
• the reviewer runs the implementer’s model. “The other tier” changes its caps only. ADR-0058 keeps it that way on public work, so no public data goes to Anthropic for review (ADR-0058).

Complexity routing for tasks: kept out of the request path

Source: agent factory design D8 and SP4 S6, §3. ADR-0047 is reserved for it and not yet written. None of it is built: on gcp-0 all 12 tasks read classifier: static, fallback: static.

What a call costs

Two price tables, and which one counts

meter.prices, USD per million tokens, list prices read 2026-10-16:

Haiku is at its over-100K-prompt rate, which an agent’s context soon reaches. Over-counting is the safe direction for a cap.

The meter’s formula

    flowchart LR
  GW["agentgateway<br/>agentgateway_gen_ai_client_token_usage"] -->|"VMPodScrape relabel<br/>input_cache_read to cached_input"| VM[("gen_ai_client_token_usage_sum<br/>by ar_agent, gen_ai_request_model,<br/>gen_ai_token_type")]
  VM -->|"meter.query, every 30 s"| M["run meter"]
  P["meter.prices<br/>keyed by served model"] --> M
  M --> F["fresh = input - cached_input - cache_creation_input<br/>cost = fresh x input + cached x cacheRead<br/>+ creation x cacheWrite + output x output"]
  F --> C1{"run cost at or over runUSD?"}
  C1 -->|"yes"| Rev["revoke: budget-run"]
  C1 -->|"no"| C2{"task cap still holds<br/>one whole run?"}
  C2 -->|"no"| Esc["escalate: budget-task"]
  F --> Met["agent_factory_run_cost_usd<br/>agent_factory_model_cost_usd_total<br/>Task status.runs.costMicroUSD"]
  

• Keyed by the served model, not the logical name: agent-default is GLM-5.3 on one listener and Opus on the other. A model with no price row is never treated as free: its run falls back to its token caps and AgentModelUnpriced fires.
• The input series includes cache tokens on Z.ai and on the non-streaming Chat Completions to Messages path, and excludes them on a streamed call or native Messages. The meter subtracts the cache tokens unless input is smaller than them, in which case it prices all of input as fresh (inputExcludesCache, agent-platform internal/factory/meter/meter.go).
• Overshoot is one scrape plus one response. The run is revoked on the meter’s next tick, and calls already in flight complete.

Worked example: a real run

Run xplane-run-jcp6kl72, an implementer on a standard task, public, served by glm-5.3, which opened cloud-native-ref#2508. Its gateway counters on 2026-10-18:

The meter recorded costMicroUSD: 1648658 on the Task: an exact match. The same run under the old token cap would have been revoked at 1.5 M of its 4.48 M tokens, with the work unfinished. Priced with no cache discount it would show as $6.39.

The same token mix on other models, as arithmetic from meter.prices and not a measurement:

With caching, Opus is not ten times GLM. Opus’s cache read ($0.20) is cheaper than GLM-5.3’s ($0.26), so on a 94%-cached run the list-price gap narrows from about 3× on input to 1.7× overall. Anthropic does bill cache writes ($5.00 per million on Opus), which this estimate leaves out. The one measured internal run, wfeiraim (triager, Opus 5.5), cost $0.477 for 303,477 input tokens, 85% of them cache reads.

Budgets: five layers, one exact

    flowchart TB
  subgraph factory["Factory: exact, enforced"]
    R["run cap: runUSD<br/>0.51 · 2.55 · 6.80"] --> TK["task cap: taskUSD<br/>1.02 · 5.10 · 13.60"]
    TK --> D["daily: factoryDaily 25M, humanDaily 5M<br/>RAW tokens"]
  end
  subgraph gateway["agentgateway: backstop, in shadow"]
    B1["B1 per run: 5M ref tokens, $8.50"]
    B2["B2 fleet: 41M ref tokens, $69.70"]
    B6["B6 Anthropic: 10M ref tokens, $17"]
  end
  L["local: 40M raw tokens per hour per sub<br/>always enforced"]
  factory --> gateway --> L
  

Efficiency

Prompt caching is the largest lever

On those 24 hours the meter’s figure is $10.40. The same tokens with no cache discount would be $41.64: caching cut the bill by 75%.

Caching is also why per-request model switching is a bad idea for agents: a new model starts with a cold cache, and on a 95%-cached run that multiplies the next call’s input cost by about 5.

Caps in USD, not tokens

The hard-task cohort found that the token cap, not the work, decided 11 of the factory’s 15 issues: it counted every cache read as a full token, and runs stopped at about 1.5M tokens and $0.58–0.76. The owner’s decision (OD-6, #2369) moved the caps to USD. Today’s live record shows the change at work:

Cheaper tiers: priced, not measured

Moving light and standard to their own names would cut the price of the same tokens to 0.11× and 0.28× (worked example above). Whether those models finish factory tasks is unmeasured. The one model comparison on record points the other way: on harder tasks the gap followed the model.

C ran B’s harness on A’s model and hit the 60-turn cap on all 15 issues, at $1.13–2.33 each. A cheaper model that fails costs more per accepted outcome than a dearer one that succeeds. The cohort’s caveat holds: fifteen issues separate only large differences, and A was stopped by its old token cap.

Self-hosted against an API

The scale-to-zero deadlock is why the floor is fixed: a model with no pod emits no metric to wake it (autoscaling).

Tuning

Every knob, what it trades and how to see the effect. Factory values live in tooling/base/agent-factory/helm-values-configmap.yaml, a gate path a maintainer changes. A meter.prices change must be mirrored in the CEL cost of infrastructure/base/agent-gateway/policy-budgets.yaml: gate AG8 fails any difference.

Move a tier to its own model:

# tooling/base/agent-factory/helm-values-configmap.yaml
tiers:
  # tier-light is glm-5.3-flash on public, claude-haiku-5-5 on internal (llm.yaml, anthropic.yaml);
  # both are priced in meter.prices, so the USD caps keep working.
  light: {model: tier-light, runTokens: 300000, taskTokens: 600000, runMinutes: 20, runUSD: 0.51, taskUSD: 1.02}

Do it with a measurement in place: run a cohort on the tier with fixed-tier arms, as the SP4 measurement rule asks, and compare accepted outcomes and USD per accepted outcome.

Change a provider model behind a name (both Z.ai backends, and the gate’s price row):

# infrastructure/base/agent-gateway/llm.yaml, on zai and zai-anthropic
modelAliases:
  tier-standard: glm-5.3-flashx   # a new model here needs its meter.prices row, or AG10 fails

In-flight runs see a mapping change on their next request. Use the factory’s pause first when that matters.

Tune the chat cascade (Semantic Router 0.2 schema):

# infrastructure/base/vllm-semantic-router/helmrelease.yaml
context_rules:
  - name: long_query
    min_tokens: 500    # raise to turn reasoning on for fewer code prompts
    max_tokens: 32000

The complexity signal and role targets need SR 0.3.0 (SP4 PR 4). Do not add a complexity rule to the 0.2 config with the documented 0.70–0.75 threshold: it never fires.

Measuring

Signal definitions and alerts are on Observability.

Known gaps