Gateways

Every model call on this platform crosses one of two gateways, chosen by who is calling. Both speak the OpenAI API; they differ in how they know the caller and in what they let it do. What runs today is on the status page. How each gateway picks a model, how to tune it and what it costs is on Model routing and cost.

Source: docs/architecture/ai-platform.drawio, page 2.

ai-gateway: humans and coding clients

The platform speaks the OpenAI API. A client points at one endpoint, names a model — or asks the platform to choose one — and never learns which pod answered.

Sending a request

# What's available
curl -sS https://llm.priv.aws.ogenki.io/v1/models \
  -H "Authorization: Bearer $LLM_API_KEY" | jq '.data[].id'

# Name a model explicitly — skips classification entirely
curl -sS https://llm.priv.aws.ogenki.io/v1/chat/completions \
  -H "Authorization: Bearer $LLM_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
        "model": "xplane-qwen-coder",
        "messages": [{"role": "user", "content": "Write a Go worker pool."}]
      }'

# Let the Semantic Router choose from the prompt
curl -sS https://llm.priv.aws.ogenki.io/v1/chat/completions \
  -H "Authorization: Bearer $LLM_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
        "model": "MoM",
        "messages": [{"role": "user", "content": "Write a Go worker pool."}]
      }'

The two calls reach the same pod. The difference is roughly 250–300 ms of classification on the MoM path — worth it when the client cannot know which model suits the prompt, wasted when it can. See Coding Clients for which client pins which model, and why.

The request path

The hop-by-hop request path shows each stage below in one picture.

Ingress. External clients arrive over Tailscale at the Cilium Gateway platform-tailscale-general and are forwarded to the Envoy AI Gateway data plane Service. In-cluster clients — OpenWebUI, the nightly Promptfoo eval — address that Service directly. Nothing is reachable from the public internet; see Private access.

Authentication. A SecurityPolicy (apiKeyAuth) targets the Gateway, so every route inherits it rather than each declaring its own. Keys come from AWS Secrets Manager through External Secrets, and the gateway strips the Authorization header before forwarding — vLLM never sees a credential.

Prompt classification (ext_proc, filter index 0). The Semantic Router is wired in as an Envoy ext_proc gRPC filter, inserted ahead of the AI Gateway’s own extproc by an EnvoyPatchPolicy — a raw xDS JSONPatch. The ordering is not optional and an EnvoyExtensionPolicy cannot express it, because that API can only append filters. The router rewrites body.model only when the client sent model: MoM (or the literal auto); an explicit xplane-* name passes through untouched.

Routing. The AI Gateway extproc derives the x-ai-eg-model header from the (possibly rewritten) body and emits gen_ai_* telemetry. An AIGatewayRoute matches that header and forwards through an AIServiceBackend → Backend → the model’s Service on port 8000.

Not every claim routes this way yet: three of the four still use a hand-written route, listed under known gaps.

Semantic routing — model: MoM

Sending model: MoM lets the Semantic Router pick from the prompt. Its decision list, highest priority first (infrastructure/base/vllm-semantic-router/helmrelease.yaml):

LoRA canaries

xplane-qwen-coder carries two LoRA adapters, each addressable as a model name of its own, and sends 10% of its traffic to one of them:

  gateway:
    enabled: true
    canaries:
      - adapter: xplane-qwen-coder-sql-dpo
        weightPercent: 10

The adapter name is matched verbatim against loraAdapters[].name — it is not derived from the base model’s name, so a typo produces a route to nothing rather than a validation error.

Canaries are mutually exclusive with the Gateway API Inference Extension’s endpoint picker. See the roadmap for where the endpoint picker stands and what turning it on would take.

Frontier models and token budgets

tier-frontier is served by GLM-5.3 through Z.ai with the platform key, which only the gateway holds. It needs no GPU, so it answers with llm-platform suspended. The ai-gateway umbrella generates its API keys in-cluster (Secret ai-gateway-api-keys in envoy-ai-gateway-system). Example (the reference host’s GKE endpoint):

curl -s https://llm.priv.gcp.ogenki.io/v1/chat/completions \
  -H "Authorization: Bearer $LLM_API_KEY" -H "Content-Type: application/json" \
  -d '{"model": "tier-frontier", "messages": [{"role": "user", "content": "Say hi"}]}'

The gateway strips x-ar-agent, x-ar-human and x-ai-gateway-client-id from every request before authentication runs, so a client-forged value never survives to be charged. Today only x-ai-gateway-client-id is then set — from the API key that matched. x-ar-agent exists only on the separate agent-router Gateway; x-ar-human arrives with ai-gateway’s oidc listener. A client cannot choose whose budget it spends.

Daily token budgets per API-key client (5M), per human (10M) and on all frontier spend (20M) are counted in shadow mode: nothing is rejected yet (ADR-0050).

The agent gateway: agent runs

Every call an agent makes, to a model, a tool or the token exchange, crosses agentgateway’s Gateway agent-router in namespace agent-gateway under the run’s own identity. The human ai-gateway stays on Envoy Gateway (ADR-0053). The harness never sees the run’s token: its identity-proxy sidecar attaches it (see Agent runtime). Manifests: infrastructure/base/agent-gateway/.

    flowchart LR
  subgraph run["Run pod, gVisor"]
    H["OpenHands harness"] --> P["identity-proxy<br/>127.0.0.1:4000 · 4002 · 4001"]
  end
  subgraph agw["agentgateway · Gateway agent-router · ns agent-gateway"]
    LP["public :8080"]
    LI["internal :8081"]
    LS["sts :8082"]
  end
  P -->|".public JWT"| LP
  P -->|".internal JWT"| LI
  P -->|"octo-sts JWT"| LS
  LP -->|"/v1, /anthropic"| Z["Z.ai GLM-5.3<br/>backends zai, zai-anthropic"]
  LP -->|"/v1, pinned alias only"| KM["Moonshot Kimi · Mistral<br/>public data only"]
  LI -->|"model route"| A["Claude<br/>Anthropic API"]
  LP -->|"/mcp"| M["agent-mcp<br/>MCP federation"]
  LI -->|"/mcp"| M
  LS -->|"/sts/exchange"| O["octo-sts"]
  M --> T["flux-operator-mcp · mcp-victoriametrics<br/>mcp-victorialogs · room-broker"]
  agw -.->|"B1, B2, B6 enforced"| RL["agent-ratelimit"] --> V[("Valkey<br/>xplane-agent-ratelimit")]
  

Listeners and data classes

Identity

Each listener then sets x-ar-agent from the verified jwt.sub, and the same sub labels every signal: x_ar_agent in the access log, ar_agent on metrics, agent.principal on spans (policies-identity.yaml, policy-frontend.yaml).

MCP federation

One AgentgatewayBackend, agent-mcp (mcp.yaml), federates the four servers with stateless sessions and names every tool <target>_<tool>: flux-operator-mcp_search_flux_docs, mcp-victoriametrics_query, room-broker_room_read. Each listener’s route carries an Allow list, one expression per role and target, so a role sees only its tools in tools/list and a denied call answers Unknown tool. A target that fails is dropped from the list rather than failing the session (failureMode: FailOpen). No run bearer reaches a server; the room-broker gets its own key in x-room-mcp-key.

The lists are generated. One profile per role and data class in infrastructure/base/agent-gateway/profiles/ names its tools, scripts/ci/gen-capabilities.py with --write renders them into mcp.yaml, and task check fails when the two differ: adding a tool is a line in a profile and its expected set in test-agent-mcp-scope (then --write).

The models are generated the same way. A profile’s models lists the aliases its role may call, and the generator renders one route-level authorization per listener into policy-capabilities.yaml, on the model routes but not /v1/models. A call whose model the caller’s profile does not name is refused 403 before any upstream call, and the access log names the run. The model is x-ar-model when the run’s identity proxy set it, else the body’s, the same choice the backends’ model transformation makes. The rule reads the raw body, so a gzip request with no x-ar-model is refused 403 too. The identity proxy sets it on every model path, over any value the harness sends: the run’s alias (the claim’s spec.model) on /v1 and /anthropic, and the completion judge’s on /judge/v1 (spec.loop.judgeModel, or none, which the rule refuses, when the claim names no judge). Every profile lists the four tier aliases, and only the public ones add the provider aliases, so on tiers the rule binds nothing until a role moves to its own size: narrowing a profile is the switch, and the CI gateway check (scripts/ci/flux-schema/assert-agent-gateway.py) refuses one that would refuse a model the factory’s tiers pin.

The arguments are generated too. A tool in a profile may read {name: <tool>, when: 'mcp.tool.arguments.<argument> not in ["<value>", ...]'}, the one form the generator takes. It becomes a route-level Require on the listener’s MCP route (capabilities-mcp-<listener>), because agentgateway 1.6’s mcpAuthorization cannot read arguments. A call that carries an excluded value, compared ASCII case-insensitively, is refused 403 before it reaches the server. The tool stays listed, and a call without the argument reaches the server, which answers it. One rule exists today: get_kubernetes_resources never with kind Secret, SecretList (the client’s list lookup resolves it to Secrets) or secrets, for every role that has the tool. It adds to the Flux MCP server’s own RBAC, which grants no Secret. The harness’s OpenHands SDK keeps its MCP session after such a 403 (since 1.55), so the run keeps its other tools.

Budgets

ADR-0050’s agent budgets, on a rate-limit server of their own so they never share a bucket with the human ones (policy-budgets.yaml, ratelimit.yaml, kvstore.yaml):

They are a backstop: the factory’s run meter stays the exact cap, and the gateway never refuses a call the factory would admit. Each call is charged at the meter’s own prices, so a cache read costs its cache rate; the CI gateway check holds the gateway’s price expression and limits to the factory’s values. They are enforced (SHADOW_MODE: "false" on the server): a call over budget gets a 429 whose body, agent_budget_exhausted, the harness logs (runbook 04, Part D). The server is envoyproxy/ratelimit (agent-ratelimit) on the Valkey KVStore xplane-agent-ratelimit; if it is down the router fails open and AgentRateLimitDown fires. A per-replica backstop of 40M raw tokens per hour per sub always enforces, above the largest run the factory allows (34M tokens of Opus cache reads for $6.80), sized so that only a runaway the run meter failed to revoke reaches it.

The token-exchange listener names each repository’s audiences, at most eight per listener; adding a repository is described on the agents overview.

What the two share, and what they keep apart

From the model routing and budgets design: