Observability

Both halves land in the platform’s Victoria stack and Grafana; nothing here runs a second telemetry pipeline. Serving is watched per model, agents per run.

    flowchart LR
  subgraph serving["Serving"]
    V["vLLM pods<br/>vllm:* :8000"]
    AIG["ai-gateway extproc<br/>gen_ai_* :1064"]
  end
  subgraph agents["Agents"]
    RUN["Agent run<br/>OTLP spans, step log"]
    AGW["agentgateway agent-router<br/>access log, gen_ai_* :15020"]
    KSM["kube-state-metrics<br/>AgentRun state"]
  end
  VM[("VictoriaMetrics")]
  VL[("VictoriaLogs")]
  VT[("VictoriaTraces")]
  COL["agent-traces-collector<br/>allowlisted metadata"]
  KEDA["KEDA<br/>three triggers"]
  G["Grafana"]
  V --> VM
  AIG --> VM
  VM --> KEDA
  RUN --> COL --> VT
  RUN -->|step log| VL
  AGW --> VL
  AGW --> VM
  KSM --> VM
  VM --> G
  VL --> G
  VT --> G
  

Serving

Both dashboards live in the Grafana folder llm (apps/base/ai/llm/grafana-folder.yaml), which only exists once the opt-in umbrella is resumed.

Alerts are lagging signals, deliberately not scale triggers:

Agents

One trace, one step log and one dashboard page per run, so a human can open any run and see where it stands, what it did, what it cost and where its time went.

Metric names. Maku’s metrics are maku_factory_* and maku_rooms_*, and the dashboards and alerts read those names.

The full transcript (prompts and outputs) stays in the room, behind room access. Alerting on runs belongs to the factory’s rules, per the observability design.

What you can inspect

A run’s page is /d/agent-run/agent-run?var-run=<id>; its This run’s window link sets the time range to the run. A resumed run counts each step once and has one trace per attempt. The room holds a run’s transcript to its end: before the harness stops, even on a reclaimed node, it asks maku-bridge to read its log to the last event.

The merge gate’s signals

The agent-factory dashboard’s Human path row reads the factory’s merge series; each series, its labels and what each decision means are on Run auto-merge safely. Cost by role reads maku_factory_run_cost_usd{tier,role}.

Alerts in observability/base/agent-platform/vmrule-agent-factory.yaml:

Spend per identity

The agent-identity dashboard (Spend per identity) shows one charged principal at a time, never All: a person or a sponsored service account. It attributes cost to the work a principal caused. It does not score the person: no panel groups, sorts or ranks by principal.

Every figure is in UTC, the ledger’s day. The sources are the factory’s per-payer series (maku_factory_charged_usd_total, the maku_factory_budget_*_usd gauges) and maku_factory_run_charge_info{run, principal, task, room}, which joins the gateway’s per-run tokens to the payer. The panels are held by test-agent-identity-dashboard and test-agent-identity-queries.

The agent gateway’s signals

agentgateway (gateways) emits under its own names. The scrape (infrastructure/base/agent-gateway/vmpodscrape.yaml) renames only the families the run meter, the budget alerts and the run page read, so they keep the gen_ai_* contract:

Every other series keeps its agentgateway_ name, and all carry namespace="agent-gateway", which keeps them apart from ai-gateway’s. Without the cache relabel the meter would price every cache read at the input rate.

Access logs are JSON with no msg, so LogsQL selects by pod labels and log.* fields:

kubernetes.pod_namespace:"agent-gateway" AND kubernetes.pod_labels.gateway.networking.k8s.io/gateway-name:"agent-router"
  | unpack_json | log.x_ar_agent:"system:serviceaccount:agents:<run>"

Useful fields: log.http.status, log.http.path, log.reason, log.route, log.x_ar_agent, log.mcp.method.name, log.gen_ai.tool.name.

Alerts in observability/base/agent-platform/vmrule.yaml, read on the Agent gateway dashboard (uid: agent-gateway):