Core metrics

  • llm.calls (count, tagged by agent + model)
  • llm.latency (histogram)
  • llm.tokens.prompt + llm.tokens.completion
  • tool.calls + tool.latency + tool.errors
  • agent.turns per invocation
  • agent.cost.usd
Advertisement

Dashboard layout

Top row: request rate, latency, error rate, cost per minute. Second row: per-tool breakdowns. Third row: per-model.

Advertisement

Alert thresholds

Cost per invocation > 2x baseline. Latency p99 > 2x baseline. Error rate > 5% sustained 5min.