Why architecture matters here
Observability architecture matters because instrumentation choices made in year one determine what you can debug in year five. Missing spans, unstructured logs, or the wrong metric cardinality leave you blind exactly when you need to see. Investing early in a coherent architecture pays back in every future incident.
Cost matters more than most teams realize. Observability data grows with traffic; a naive setup costs 10-20% of infra spend. Sampling, cardinality control, retention tiers, and log volume management all belong in the architecture — not addressed after the bill arrives.
Reliability of observability is a real constraint. When your prometheus is down at the same time as your service, you have two problems. Multi-region observability, careful sampling, and offline aggregation keep the truth flowing even during outages.
The architecture: every layer explained
Walk the diagram top to bottom.
Instrumented Apps. OpenTelemetry SDKs (or auto-instrumentation agents) emit spans, metrics, and logs. Structured logs with a correlation ID. Metrics with intentional labels (avoid unbounded cardinality). Spans on every meaningful boundary.
Sidecars / Infra. Service mesh sidecars emit L7 metrics and traces without app changes. Kubelet + node exporter emit system metrics. eBPF probes emit kernel-level metrics without instrumentation.
OTel Collector. The central pipeline. Receives OTLP over gRPC/HTTP, applies processors (batching, sampling, redaction, transformation), and exports to backends. Running the collector as a fleet (daemonset + gateway) gives resilience and central control.
Metrics Backend. Prometheus is the OSS default. At scale, Cortex, Mimir, or Thanos federate multiple Prometheus into a global view. Commercial (Datadog, New Relic) handle scale as a service.
Traces Backend. Tempo (Grafana), Jaeger, or commercial APM. Trace storage is expensive; head-based or tail-based sampling controls cost. Retention typically 3-30 days.
Logs Backend. Loki for label-driven cheap logs; Elastic for full-text search; commercial (DD Logs, Splunk) for higher-priced managed. Log volume dominates observability cost.
Continuous Profiling. Parca or Pyroscope collect stack samples continuously. Flamegraphs at your fingertips reveal CPU hotspots without waiting for an incident.
eBPF Telemetry. Cilium Hubble, Pixie, and similar tools use eBPF to get kernel-level visibility. Network flows, syscall latencies, and process events without app changes.
Alertmanager + SLO Engine. Alerts route to on-call. Multi-window burn-rate alerts on SLOs give the right density: page for real problems, silence for noise.
Dashboards + Runbooks. Grafana dashboards linked to runbooks. Each alert has a runbook URL; on-call opens it, follows steps, resolves.
On-call platform. PagerDuty, Opsgenie, or similar handles escalation. Slack channels for real-time coordination. IR playbook lives in a wiki with quarterly drills.
End-to-end incident trace
Trace an incident. A user reports checkout is slow. The on-call opens the dashboard for the checkout service. p99 latency is up 3x in the past 20 minutes. Error rate is normal.
The dashboard has a link to traces. On-call opens a slow trace: the outbound call to the payment service is taking 2 seconds instead of 200 ms. Payment service dashboard shows nothing unusual — its metrics are normal.
On-call opens Pyroscope for the payment service. A new deploy went out 30 minutes ago. Flamegraph shows a new hot method taking 60% of CPU. The change is on the previous commit.
On-call decides to roll back. One click; the previous deploy comes back. Metrics recover within a minute. Total time from page to resolution: 15 minutes. Written up as a post-mortem; the runbook and the alert are updated.
Without any single layer of the stack this would have taken hours: without traces, you would not have found the payment service link; without profiling, you would not have identified the hot method; without dashboards linked to runbooks, on-call would have been rooting through documentation. Each layer contributes.