Serving

An OpenAI-compatible inference platform on EKS: vLLM on L4 spot GPUs, fronted by Envoy AI Gateway, scaled by KEDA on vLLM saturation signals, and declared as a single Crossplane InferenceService claim per model.

This platform is off by default. Three independent gates must all be released before anything LLM-related exists on the cluster — see Turning it on. A plain terramate script run deploy and a plain Flux reconciliation both leave the cluster LLM-free.

At a glance


Engine	vLLM, one Deployment per model, port 8000
Gateway	Envoy Gateway + Envoy AI Gateway `1.1.0`
Routing	`AIGatewayRoute`, keyed on the `x-ai-eg-model` header
Prompt routing	vLLM Semantic Router, as a gRPC `ext_proc` filter — only acts on `model: MoM`
Autoscaling	KEDA, three vLLM saturation triggers OR-combined, `min=1` (always warm)
Weights	Amazon S3 Files (POSIX over S3), RWX PVC shared by a preload Job and the serving pod
GPUs	Karpenter `gpu-l4` NodePool — single-GPU `g6` spot-first instances, Bottlerocket NVIDIA AMI, capped at 4 GPUs
Composition	Inline KCL in crossplane-configuration (`apis/inferenceservice/`), shipped in the `crossplane-configuration-aws` package the host pins

Turning it on

Example (the reference host): the paths below are its EKS cluster’s, aws-0. The AWS gate and the two Kubernetes gates are independent of each other, so releasing one does not bring the others along — but llm-platform itself dependsOn ai-gateway, so the second Kubernetes command must run before the third:

# Gate 1 — AWS side (S3 Files filesystem + IAM). Terramate stack tagged
# `opt-in`; skipped unless TM_LLM_PLATFORM_ENABLED=true (verified in
# opentofu/aws/llm-platform/workflows.tm.hcl — unset or != "true" echoes [skip]
# and exits 0).
TM_LLM_PLATFORM_ENABLED=true terramate -C opentofu/aws/llm-platform script run deploy

# Gate 2 — Kubernetes side, gateway layer: already released
# (clusters/aws-0/ai-gateway.yaml, spec.suspend: false). llm-platform depends on it.

# Gate 3 — Kubernetes side, GPU models. The umbrella Flux Kustomization ships
# suspended (spec.suspend: true, clusters/aws-0/llm-platform.yaml).
flux resume kustomization llm-platform -n flux-system

The umbrella aggregates 5 child Flux Kustomizations under clusters/aws-0-llm-platform/:

Child	Renders	Path
`runtimeclass-nvidia`	`RuntimeClass nvidia`	`infrastructure/base/runtimeclass-nvidia`
`llm-platform-gpu-nodepools`	Karpenter `gpu-l4` NodePool + EC2NodeClass	`infrastructure/base/karpenter-nodepools-gpu`
`llm-platform-apps`	The `InferenceService` claims + OpenWebUI	`apps/llm`
`llm-platform-security-epi`	The preload Job’s EKS Pod Identity	`security/base/epis-llm`
`llm-platform-promptfoo`	Nightly agent-eval CronJob	`tooling/base/promptfoo`

The gateway layer these children attach to (Envoy Gateway, the Envoy AI Gateway, the Semantic Router and the ai-gateway Gateway) is a separate umbrella, ai-gateway, under clusters/aws-0-ai-gateway/. It is CPU only and released by default, since the Agent Factory needs it; llm-platform depends on it.

That directory is a sibling of clusters/aws-0/, not a child, on purpose: flux-system syncs clusters/aws-0/ recursively, so a nested path would be auto-discovered and applied — bypassing the suspend gate entirely.

On GKE

Example (the reference host): its GKE cluster is gcp-0. GKE has no OpenTofu gate: the weights bucket is a Crossplane claim rather than an OpenTofu stack, so there is no TM_LLM_PLATFORM_ENABLED. The two Kubernetes gates match EKS’s: clusters/gcp-0/ai-gateway.yaml, then clusters/gcp-0/llm-platform.yaml, which depends on it. The first is released (spec.suspend: false, for the Agent Factory); the second ships spec.suspend: true. Weights are served from a GCS bucket over the Cloud Storage FUSE CSI driver instead of an S3 Files POSIX mount — see ADR-0021 for why, including what it gives up.

On GKE the umbrella stays suspended on cost and GPU quota, not on missing identity: each claim’s per-claim GCP read identity is rendered by the composition. Check the project’s GPUS_ALL_REGIONS quota before you resume it: a new project’s is 0, and no claim can schedule a GPU until Google raises it. See clusters/gcp-0-llm-platform/README.md (example path) for the failure-order watch list before a resume.

Security posture

Zero trust by default. Every workload carries a default-deny CiliumNetworkPolicy. The serving pod’s egress is kube-dns only; only the bounded preload Job is granted world:443, because it is short-lived and the serving pod cannot reach HuggingFace even in principle.
No credentials in Git. API keys and the HuggingFace token come from AWS Secrets Manager through External Secrets — see PKI & Secrets.
Read-only IAM on the serving pod. Each claim’s serving ServiceAccount carries a per-claim EKS Pod Identity scoped to read its own weights prefix, rendered by the composition; only the shared preload Job’s identity can write to the bucket.
Private ingress only. Reachable exclusively from the tailnet — see Private Access.

Known gaps

xplane-llamaguard3-1b holds a GPU and serves no automatic traffic — it runs at min=1 but appears in no Semantic Router decision rule.
Gateway routing is half-migrated — only xplane-qwen-coder is composition-owned; the other three claims still route through the hand-written apps/base/ai/llm/ai-gateway-routes/route.yaml, so adding a model means adding its route by hand unless the claim opts in (spec.gateway.enabled: true).
The Gateway API Inference Extension’s endpoint picker is implemented but enabled on zero claims. It is mutually exclusive with LoRA canaries, and the only gateway-enabled claim uses a canary.
No distributed tracing. OTLP export from the AI Gateway extproc is written but not enabled, pending verification against VictoriaTraces.

What comes next for serving is on the roadmap.

The InferenceService claim

One model, one YAML file — a complete claim with its reasoning intact, what it renders, and every field it accepts.

Autoscaling & GPUs

Three KEDA triggers on leading vLLM signals, the scale-to-zero deadlock, the gpu-l4 NodePool and S3 Files weights.