All 54 articles, sorted alphabetically
Alert Fatigue Solutions, in depth: measuring noise from alert history, six noise signatures and the targeted fix for each
A data-driven approach to alert fatigue: build an episode table from alert history, compute flap rate, resolve-before-ack, co-firing, low-traffic spik…
Read article →Alerting That Doesn't Burn Out Oncall, in depth: page budgets, outcome records, per-rule precision and the weekly review that deletes alerts
How to run alerting as a measured product so on-call stays sustainable: set a page budget, record the outcome of every page, compute per-rule precisio…
Read article →Chaos Engineering and Observability, in depth: steady state as SLI queries, experiment tagging, burn-rate abort guards and testing your alerts
How to wire chaos experiments into observability: steady-state hypotheses written as SLI queries, tagging experiments into metrics, traces and dashboa…
Read article →Distributed Tracing with Jaeger, in depth: the v2 collector architecture, storage, Kafka buffering, remote and adaptive sampling, and running it in production
How Jaeger v2 works and how to run it: one binary built on the OpenTelemetry Collector in collector, query, ingester and all-in-one roles, choosing tr…
Read article →Golden Signals Revisited, in depth: measuring latency, traffic, errors and saturation correctly for queues, streaming and autoscaled services
The four golden signals re-examined for modern systems: what the SRE book actually prescribed, where to measure, latency split by outcome, traffic in …
Read article →Grafana
A deep dive into Grafana's architecture, capabilities, and ecosystem: multi-source querying, Loki for logs, Tempo for traces, dynamic dashboardin…
Read article →Loki vs Elastic vs ClickHouse for Logs
A deep comparison of Loki, Elasticsearch, and ClickHouse as a log store: the three indexing philosophies, the cardinality story that separates them, s…
Read article →Metric Cardinality Management, in depth: measuring series, setting budgets, enforcing limits at every hop and redesigning labels that explode
How to run metric cardinality as an ongoing programme: count series by metric and label with the Prometheus TSDB status API and PromQL, attribute them…
Read article →Adaptive trace sampling architecture
Deep-dive on adaptive sampling for distributed tracing: why fixed-rate sampling fails, a rate governor that measures per-operation throughput and allo…
Read article →Alerting Architecture, in depth: rule evaluation, Alertmanager routing, grouping, inhibition, high availability and testing
How an alert travels from a metric to a person: rule evaluation with for and keep_firing_for, the ALERTS series, Alertmanager's routing tree, gro…
Read article →Metrics anomaly detection architecture
Deep-dive on anomaly detection for metrics at scale: series routing and windowing, seasonal baseline stores, cheap detector ensembles with persistence…
Read article →Burn-rate SLO alerting architecture
Deep-dive on multi-window multi-burn-rate SLO alerting: why a burn rate (observed error ratio divided by the budget the SLO permits) turns an alert in…
Read article →Metric cardinality -- the silent cost driver of observability
Deep-dive on metric cardinality: every label combination as a distinct time series, cardinality as the product of label value counts, explosions from …
Read article →Dashboards-as-code architecture
Deep-dive on building observability dashboards as versioned code: shared panel and query libraries for consistency, CI validation that catches broken …
Read article →eBPF observability architecture, in depth: hooks, maps, in-kernel aggregation, CO-RE and running agents in production
How eBPF observability works from event to dashboard: choosing tracepoints, kprobes, fentry or uprobes, ring buffers versus in-kernel maps, a CO-RE sy…
Read article →Metrics Exemplars, in depth: linking a latency histogram to the exact trace behind a spike
How metrics exemplars work end to end: the OpenMetrics exposition format, attaching trace ids in Go, Python and OpenTelemetry, exemplar filters and re…
Read article →Full-Stack Observability Architecture in Depth
Full-stack observability architecture: what each layer emits, trace-id correlation, agent and gateway collectors, tail sampling, cost modelling, SLO b…
Read article →Grafana, in depth: how a panel becomes queries, why dashboards get slow, and how to run Grafana as code
How Grafana turns a panel into data source queries: max data points, $__interval, $__rate_interval, variables, repeats and collapsed rows, transformat…
Read article →Head-based trace sampling architecture
Deep-dive on head-based sampling for distributed tracing: the root-span sampling decision, the sampled flag in propagated trace context, consistent tr…
Read article →Health check architecture
Deep-dive on health checking: why liveness, readiness, and startup probes ask different questions, shallow vs deep checks, how a shared /health endpoi…
Read article →Latency histograms and quantile estimation architecture
Deep-dive on latency histograms: hot-path bucket counters, why percentiles don't average, cumulative buckets and mergeability, query-time quantil…
Read article →Jaeger, in depth: the read side of tracing, how spans are indexed, what search can and cannot find, the critical path, the api_v3 query API and the Monitor tab
How Jaeger turns stored spans into answers: the span data model, how Cassandra and Elasticsearch index traces, two-phase search and its blind spots, a…
Read article →Logs architecture, in depth: structured events, shippers, buffers, indexes, retention tiers and the cost model
How a production log pipeline works end to end: what a log event should contain, how node shippers tail and checkpoint files, where to parse, redact a…
Read article →Metric downsampling architecture
Deep-dive on metric downsampling: why lossy aggregation must preserve the extremes incidents live in, the tiered storage model and cascading rollups, …
Read article →Metrics architecture
Deep-dive on metrics architecture: instrumentation, TSDB, counters/gauges/histograms, cardinality budgets, recording rules, alerts.
Read article →OpenTelemetry, in depth: the API and SDK split, providers, processors and exporters, resources, semantic conventions, OTLP and configuration
How OpenTelemetry is built and how to use it well: why the API is separate from the SDK, what tracer, meter and logger providers, samplers, span proce…
Read article →OpenTelemetry Collector architecture
Deep-dive on the OTel Collector: receiver/processor/exporter pipelines and connectors, daemonset agents vs gateway fleets, trace-affinity tail samplin…
Read article →Continuous profiling architecture, in depth: sampling, stack unwinding, symbolization, storage and diff flame graphs
How a continuous profiler works end to end: sampling statistics and rate choice, frame-pointer versus DWARF unwinding, off-host symbolization by build…
Read article →Prometheus, in depth: the life of a sample on one server, from exposition and the scrape loop to the TSDB head, compaction and query evaluation
How one Prometheus server really works, followed one sample at a time: the exposition format and instrumentation, discovery and the two relabel stages…
Read article →Recording rules architecture
Deep-dive on monitoring recording rules: evaluating expensive queries on a fixed interval and storing results as cheap derived series, the rule evalua…
Read article →Real User Monitoring
Deep-dive on Real User Monitoring: browser SDK instrumentation, Core Web Vitals (LCP/INP/CLS), session and error tracking, beacon transport, field vs …
Read article →OpenTelemetry semantic conventions architecture
Deep-dive on OpenTelemetry semantic conventions, the standardized registry of attribute names, types, and values that makes telemetry correlatable acr…
Read article →SLO Engineering Architecture in Depth: choosing SLIs, error-budget arithmetic, SLOs as code and the policy that makes them bite
How to build an SLO system end to end: picking SLIs and where to measure them, budget math with worked numbers, latency SLIs from histograms, SLOs as …
Read article →SLOs and SLIs, in depth: an SLI for every system shape, from pipelines and queues to storage and scheduled jobs, and how to prove it tracks user pain
How to define and implement service level indicators for systems that are not request and response: freshness and coverage for data pipelines, timely …
Read article →Span links, in depth: connecting traces across batches, fan-in, retries and async hops
How OpenTelemetry span links work and when to use them instead of a parent: the Link data model, batch consumers with one link per message, Kafka code…
Read article →Span metrics architecture
Deep-dive on span metrics: the OTel Collector spanmetrics connector, RED metrics (rate/errors/duration) with dimensions, exemplars linking metrics to …
Read article →Synthetic monitoring -- testing from the outside, always
Deep-dive on synthetic monitoring: proactive scripted probes (uptime, API, browser flows), running from many locations, the comparison to real user mo…
Read article →Tail-based trace sampling architecture
Deep-dive on tail-based sampling: why head sampling loses incident evidence, trace-ID-aware load balancing, decision buffers and windows, keep policie…
Read article →The Three Pillars of Observability, in depth: what metrics, logs and traces each answer, how to join them, what they cost, and where the model falls short
Metrics, logs and traces explained as one system: the question each answers, their data models and cost drivers, one request emitting all three, joini…
Read article →Trace context propagation architecture
Deep-dive on distributed trace context propagation: the W3C traceparent wire format, inject/extract at every boundary, parent/child span linkage, bagg…
Read article →Distributed tracing architecture
Deep-dive on distributed tracing: spans, W3C context propagation, head + tail sampling, OTel Collector, storage, and correlation with logs and metrics…
Read article →Observability, in depth: a system property you can test, wide events, high-cardinality data and the investigation loop
Observability as a property of a system rather than a product: the control-theory definition, how it differs from monitoring, a five-question test, wi…
Read article →On-Call Runbooks
Write and maintain runbooks for every alert: structure, examples, testing, escalation. Turn incident knowledge into executable playbooks.
Read article →OpenTelemetry Full Stack, in depth: one trace from the browser through services, Kafka and Postgres, with agent and gateway collectors
How to instrument a whole system with OpenTelemetry so one request can be followed end to end: resource identity, browser tracing and CORS, zero-code …
Read article →OpenTelemetry Pipeline Design, in depth: buffers, backpressure, durable queues, sizing and loss accounting
How to design an OpenTelemetry pipeline that survives production: where data waits and where it is dropped from SDK to backend, processor ordering, me…
Read article →Blameless Postmortem Culture, in depth: what blameless means, just culture and accountability, leadership behaviour, incentives, evidence from telemetry and measuring safety
How to build and keep a blameless postmortem culture: what blameless does and does not mean, just culture and forward-looking accountability, why blam…
Read article →Continuous Profiling in Production, in depth: choosing a collection model, per-runtime setup, overhead budgets, container permissions and using profiles in incidents
Running continuous profiling safely in production: what it adds over one-off profiles, SDK push versus eBPF agents, setup for Go, the JVM, Python and …
Read article →Prometheus at Scale, in depth: measuring limits, cutting cardinality, sharding, HA pairs, remote write and the Mimir and Thanos backends
A decision path for scaling Prometheus: the limits of one server and how to measure them, cardinality controls, functional and hashmod sharding, HA pa…
Read article →Prometheus
Pull-based metrics collection, time-series storage, PromQL query language, metric types (counter, gauge, histogram, summary), service discovery, alert…
Read article →RED Method vs USE Method, in depth: what each measures, PromQL for both, software resources, a worked incident and how to combine them
RED (rate, errors, duration) and USE (utilization, saturation, errors) compared from first principles: which questions each answers, RED in PromQL fro…
Read article →SLIs, SLOs, and Error Budgets, in depth: measuring the right events, building the ratio and keeping a budget ledger
A first-principles guide to SLIs, SLOs and error budgets as a measurement and accounting problem: where to measure, what counts as valid and good, req…
Read article →SLI, SLO and SLA Explained, in depth: from a measured indicator to a contract you can afford
A first-principles guide to the three reliability layers: what SLIs, SLOs and SLAs are for and who owns them, downtime arithmetic, the clauses of a re…
Read article →Structured Logging Best Practices, in depth: event schemas, context propagation, trace correlation, redaction and volume control
How to emit logs that machines can query and humans can trust: events as typed records, a shared schema and naming convention, level policy, request c…
Read article →Trace Sampling Strategies, in depth: head, tail, consistent probability and hybrid pipelines, and how to keep counts honest
How to choose and combine trace sampling strategies: what each decision point can see, head sampling with ParentBased samplers, consistent probability…
Read article →