Why it matters
Without observability, agents are black boxes. When users complain, you can't reproduce or diagnose. With good instrumentation, you can trace exactly what happened.
The architecture
OpenTelemetry: standard for traces + metrics. Every agent step becomes a span; spans nest to show turn structure.
Spans: agent turn (outer), model call (inner), tool call (inner). Tags include model, tool name, token counts, latency.
How it works end to end
Metrics: request rate, error rate, p50/p95/p99 latency, token count per turn, tool call count, cost estimate.
Structured logs: JSON logs with correlation IDs. Include session ID, user ID, request ID.
Tracing sensitive data: model inputs/outputs can contain PII. Redact before shipping to observability platform.
Cost tracking: token count × price per token = cost per request. Track for cost accounting.