Full-Stack Observability Architecture in Depth

Signals and what each layer should emit

Each signal answers a different question, and confusing them is the root of most cost and blind-spot problems. Metrics are pre-aggregated numbers over time: cheap to store and query, perfect for alerting and trends, useless for explaining one specific request. Traces record the path and timing of individual requests as trees of spans; they explain where time went, but you can afford to keep only a sample. Logs are timestamped events with arbitrary detail; they explain why a branch was taken, and they are usually the largest line on the bill. Continuous profiles sample stacks at a low rate and show which code burns CPU or allocates memory, which no other signal can.

The rule of thumb: alert on metrics, investigate with traces, explain with logs, optimise with profiles. If a question is asked every day, answer it with a metric; if it is asked once per incident, a trace or a log is fine.

Signals and what each layer should emit

Each signal answers a different question, and confusing them is the root of most cost and blind-spot problems. Metrics are pre-aggregated numbers over time: cheap to store and query, perfect for alerting and trends, useless for explaining one specific request. Traces record the path and timing of individual requests as trees of spans; they explain where time went, but you can afford to keep only a sample. Logs are timestamped events with arbitrary detail; they explain why a branch was taken, and they are usually the largest line on the bill. Continuous profiles sample stacks at a low rate and show which code burns CPU or allocates memory, which no other signal can.

The rule of thumb: alert on metrics, investigate with traces, explain with logs, optimise with profiles. If a question is asked every day, answer it with a metric; if it is asked once per incident, a trace or a log is fine.

Correlation: one trace id across every layer

Signals from different layers are only useful together if they share keys. Two kinds of key do the work. Resource attributes identify the emitter: service.name, service.version, deployment.environment, k8s.pod.name. Set them once in the SDK or let the collector add them, and use exactly the same names in every backend, or the pivot from a metric to the pods behind it breaks. Trace context identifies the request. The W3C Trace Context header carries it between processes:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             ^^ ^-------- 32 hex trace id -------^ ^-- span id --^ ^^ flags (01 = sampled)

Every service extracts the header, starts a child span, and injects a new header on outgoing calls, including into message headers on queues, which is the hop teams most often forget. Then the same trace id goes into every log line written while the span is active, and metric exemplars attach sample trace ids to histogram buckets. The result is a three-way pivot: from a latency spike on a dashboard, click an exemplar to the slow trace, then from a span to the logs carrying its trace id.

# Python: put the active trace id into every structured log record
import logging
from opentelemetry import trace

class TraceIdFilter(logging.Filter):
    def filter(self, record):
        ctx = trace.get_current_span().get_span_context()
        record.trace_id = format(ctx.trace_id, "032x") if ctx.is_valid else ""
        record.span_id = format(ctx.span_id, "016x") if ctx.is_valid else ""
        return True

Propagation details, baggage and the async edge cases are in trace propagation.

The collection pipeline: agents and gateways

Applications should not talk to backends directly. They send OTLP to a collector, which lets you change vendors, sampling and redaction without redeploying code. The common shape has two tiers. An agent collector runs on every node (a DaemonSet in Kubernetes): it receives from local pods, enriches with Kubernetes metadata, batches, and protects itself with a memory limiter. A gateway tier receives from agents and does the expensive work: tail sampling, redaction of sensitive attributes, routing to backends.

Tail sampling has a hard requirement that shapes the topology: every span of a trace must reach the same gateway instance, because the decision needs the whole trace. So the agent tier uses a load-balancing exporter keyed by trace id in front of the gateways. The gateway then holds spans in memory for a decision window and keeps errors, slow traces and a baseline sample.

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20
  tail_sampling:
    decision_wait: 10s        # hold spans this long before deciding
    num_traces: 100000        # traces held in memory at once
    policies:
      - name: keep-errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: keep-slow
        type: latency
        latency: {threshold_ms: 500}
      - name: baseline
        type: probabilistic
        probabilistic: {sampling_percentage: 5}
  batch: {}
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [otlp/traces]

Size the gateway for the decision window: spans per second times decision_wait times average span size is the memory the sampler needs. At 50,000 spans per second, 10 seconds and 1 KB per span that is about 500 MB per replica set before overheads. Tail sampling covers the policy options in more depth.

Cost modelling and storage tiers

Observability cost scales with traffic, and it is easy to let it grow faster than the product. Model it before you pay for it. The drivers are different per signal: metrics cost by active series, traces by spans kept, logs by bytes ingested and retained.

Metrics cardinality is the trap that bites first: series count is the product of the label values, so a histogram with 12 buckets, 40 routes, 5 status classes and 300 pods is 720,000 series by itself. Put unbounded values such as user ids and raw URLs in traces and logs, never in metric labels. Cardinality management has the auditing techniques.

Tier retention by how old data is actually used. Keep raw metrics for weeks and downsampled rollups for a year; keep traces for one or two weeks; keep logs searchable for days and archived to object storage for compliance. Most queries touch the last 24 hours.

Finally, attribute the bill. Tag every signal with the owning team through resource attributes, publish a monthly cost per team and per signal, and give each team a budget for series and log bytes. Teams that can see their own telemetry spend usually cut it without being asked, and the platform team stops being the only group arguing for restraint. Review the top ten metric names and log sources by volume every quarter; a handful of noisy emitters almost always account for most of the growth.

Closing the loop: SLOs and burn-rate alerts

Observability earns its keep when it pages the right person for the right reason. Alert on symptoms users feel, defined as service level objectives, rather than on causes such as CPU. A 99.9 percent availability SLO over 30 days gives an error budget of 0.1 percent, about 43 minutes of full outage. Burn rate is how fast you are spending it: a burn rate of 1 uses exactly the budget over the window, and 14.4 uses 2 percent of a 30-day budget in one hour. The SRE workbook pattern pages when both a long and a short window exceed the threshold, so the page fires fast and also stops fast once the problem is fixed.

# page: 2% of the 30-day budget burned in 1h, still burning in the last 5m
(
  sum(rate(http_requests_total{job="checkout",code=~"5.."}[1h]))
  / sum(rate(http_requests_total{job="checkout"}[1h]))
) > (14.4 * 0.001)
and
(
  sum(rate(http_requests_total{job="checkout",code=~"5.."}[5m]))
  / sum(rate(http_requests_total{job="checkout"}[5m]))
) > (14.4 * 0.001)

Pair it with a slower page (burn rate 6 over 6 hours and 30 minutes) and a ticket for slow leaks (burn rate 1 over 3 days and 6 hours). Burn-rate alerting walks through the maths. Every alert links a runbook and a dashboard that starts at the SLO and drills down.

Worked example: a nine-minute incident

Checkout latency SLO: 99 percent of requests under 800 ms. At 14:02 the fast burn alert fires. The on-call engineer opens the checkout SLO dashboard: p99 jumped from 420 ms to 2.1 s, error rate flat, and only the version 3.18 pods are affected. An exemplar on the slow histogram bucket links to a trace. The trace shows 1.7 s in a single span: a client span from checkout to pricing, and inside pricing a database span running a query without the expected index hint. Pivoting on the trace id into the logs shows the pricing service logged a cache miss for every request, and a deploy event at 13:58 changed the cache key format. The continuous profile for pricing confirms CPU shifted into query serialisation. Rollback at 14:11, burn rate back under 1 by 14:20. Nine minutes from page to mitigation, with no guesswork, because version was a resource attribute, exemplars were on, and logs carried trace ids.

Failure modes

• Broken context. One hop drops the traceparent header (an async queue, a hand-rolled HTTP client) and every trace splits in two. Check the share of root spans per service; a spike means propagation broke.
• Fragmented tail sampling. Spans of one trace land on different gateways and the sampler keeps half a trace. Use trace-id-keyed load balancing in front of samplers.
• Cardinality explosion. A new label with request ids takes the TSDB down. Enforce label allow-lists in the collector and alert on series growth.
• Observability shares fate with production. The monitoring stack runs in the cluster it watches and dies with it. Run alerting and a minimal metrics path outside the blast radius, and monitor the monitors from a second location.
• Backpressure into the app. A slow backend fills the SDK export queue. SDKs should drop, not block; watch the dropped-span and refused-data metrics the SDKs and collectors expose.
• Sensitive data in telemetry. Tokens and personal data leak into attributes and logs. Redact at the gateway, and treat the telemetry store as production data.

What to do next

1. Standardise resource attributes and set service.name and version on every workload.
2. Verify trace context crosses every hop, including queues, by checking root-span ratios per service.
3. Add trace and span ids to every structured log line and turn on exemplars.
4. Deploy agent plus gateway collectors with memory limiters and trace-id load balancing before tail sampling.
5. Build a cost model per signal and set a cardinality budget per team.
6. Define SLOs for your top three user journeys and replace cause-based pages with multi-window burn-rate alerts.
7. Run the alerting path outside the production cluster and test it in a game day.