OpenTelemetry Collector architecture — pipelines, agent/gateway tiers, tail sampling

Why architecture matters here

The collector exists because the alternative — applications exporting directly to backends — welds policy into every deployed binary. Change vendors, add a destination, tighten PII redaction, adjust sampling: each becomes a coordinated redeploy of every service in the company. The collector inverts the coupling: applications speak OTLP to a local endpoint and are done; everything that will change — destinations, credentials, sampling rates, enrichment, cost controls — lives in pipeline configuration owned by the observability team. It is the same argument as message brokers and API gateways: an indirection layer at exactly the seam where change is constant.

The economics run deeper than convenience. Telemetry is a firehose priced by the drop: traces at production volume can cost more than the compute they observe, and the highest-leverage controls — tail-based sampling (keep every error trace and latency outlier, 1% of the boring rest), metric aggregation and filtering, log routing by value tier (errors to the expensive indexed store, debug to cheap object storage) — are only implementable in a layer that sees complete signals before the vendor's meter runs. A gateway fleet with tail sampling routinely cuts trace spend 10× while improving incident coverage, because the 1% it keeps is chosen, not random.

And the collector is where compliance meets telemetry physically. Attributes carry PII (URLs with emails, headers with tokens) that must not reach third-party backends; data-residency rules may require EU telemetry to stay in EU backends; audit requires knowing what left. OTTL-based redaction, attribute allowlists, and region-aware routing in the gateway tier are enforceable, auditable policy — versus hoping every team's instrumentation remembered. The collector is not plumbing; it is the policy engine for the most privacy-dense data stream your infrastructure produces.

The architecture: every piece explained

Top row: the component model. Receivers ingest: OTLP gRPC/HTTP (the native protocol from SDKs and other collectors), a full Prometheus scraper (the collector can replace Prometheus agents), filelog tailing with parsers, Kafka, cloud-provider receivers, and dozens more from the contrib registry. Processors run in declared order per pipeline: memory_limiter first (refuse data before OOM), then transforms/filters/enrichment, with batch last-ish (compression efficiency for export). Exporters deliver: OTLP to vendors or upstream collectors, Prometheus remote-write, Kafka, file/debug — and a pipeline may fan out to several (the migration pattern: dual-write old and new vendor for a quarter). Pipelines wire it per signal type; the same receiver can feed multiple pipelines, and connectors join them: the spanmetrics connector consumes the trace pipeline and emits RED metrics into the metrics pipeline — service dashboards derived from traces with zero extra instrumentation.

Middle row: topology and the hard feature. Agent mode puts a collector on every node (daemonset) or beside every pod (sidecar): apps export over localhost (no network loss window), and the agent enriches with what only locality knows — k8s metadata (pod/namespace/node via the k8sattributes processor), host resources. Gateway mode is the central fleet behind a load balancer: fewer, bigger collectors owning policy that requires the full stream. Tail sampling is the gateway's marquee job and its scaling constraint: to decide 'keep this trace?' on error/latency criteria you must buffer complete traces — which means all spans of a trace must reach the same gateway instance. The pattern is two-layer: a first tier load-balances by trace_id (the loadbalancing exporter), a second tier buffers (seconds of spans in memory) and applies policies (error traces: always; p99 latency: always; the rest: probabilistic 1%). Memory limiter and sending queues are the self-protection kit: soft/hard memory thresholds that trigger refusal (backpressuring sources) before the kernel's OOM killer makes the decision, and per-exporter persistent queues that ride out backend outages.

Bottom rows: the transformation language and the ops reality. OTTL (the OpenTelemetry Transformation Language) powers the transform processor: redact (replace_pattern(attributes["url"], ...)), enrich, rename, drop attributes, reshape spans/metrics/logs — the policy vocabulary, applied declaratively. The ops strip is recursive but essential: the collector exports its own telemetry (accepted/refused/dropped per component, queue depths, memory), and the observability team's first dashboard is the pipeline's; config ships as code through CI with validation; capacity is planned per signal per tier.

End-to-end flow

Follow telemetry through a two-tier estate during a normal day and a bad one. A checkout pod emits spans via OTLP to its node's agent (localhost, 1ms). The agent's pipeline: memory_limiter → k8sattributes (annotates pod, deployment, node) → resource detection → batch → OTLP exporter to the gateway VIP. Metrics and logs ride parallel pipelines on the same agent — the Prometheus receiver scrapes the pod's endpoints; filelog tails container logs with parsing to structured records.

At the gateway, tier one hashes each span's trace_id via the loadbalancing exporter to tier two, where the tail_sampling processor buffers traces for 10 seconds and applies policy: this trace touched an error? Keep. Duration > 2s? Keep. Contains a span from the payments service (compliance requirement)? Keep. Otherwise: 1% lottery. Kept traces flow through an OTTL transform that redacts card-number-shaped strings and strips two internal-only attributes, then export: the APM vendor (primary) and a Kafka topic feeding the data lake (long-term). The spanmetrics connector meanwhile emits per-service RED metrics from all spans (pre-sampling — the metrics see 100% even though only 1.4% of traces persist), which is how dashboards stay accurate over sampled traces. Net effect on the bill: 98.6% fewer trace ingests, zero lost error visibility.

The bad day: the APM vendor's ingest has a 40-minute outage. Gateway exporters hit failures; persistent sending queues absorb — disk-backed, sized for an hour at peak — while the Kafka export continues unaffected (independent queues per exporter). Queue-depth alerts fire (the pipeline's own telemetry); nobody loses data; when the vendor recovers, queues drain with rate limiting so the backlog doesn't DDoS the recovered ingest. Two weeks later, the same machinery earns its keep differently: the company adds a second APM vendor for evaluation — one exporter block and a 5% routing rule in the gateway config repo, reviewed, merged, live in an afternoon; not one application team was involved. And when security asks 'can any telemetry contain auth tokens?', the answer is an OTTL rule and its test suite in git — a pull request, not an archaeology project across three hundred services.

What is the OpenTelemetry architecture, end to end

The OpenTelemetry architecture has three layers: instrumentation inside each process (an API plus an SDK), a wire protocol called OTLP, and an optional Collector that receives, processes and exports telemetry to one or more backends. OpenTelemetry stores nothing itself; storage, querying and alerting belong to the backend you export to.

The Collector part of that picture is one YAML file. A minimal agent that accepts OTLP from local SDKs and forwards traces and metrics to a gateway looks like this:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20
  batch: {}
exporters:
  otlp_grpc:
    endpoint: otel-gateway.observability.svc:4317
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp_grpc]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp_grpc]

Declaring a component does not switch it on: only components listed in a pipeline under service run. Processors execute in the order listed, which is why memory_limiter comes first, so it can refuse data before the process runs out of memory. Collector 0.162.0 names the gRPC exporter otlp_grpc; older releases call it otlp, which 0.162.0 still accepts with a deprecation warning. Check a file before deploying it with otelcol validate --config=agent.yaml. How the API and SDK fit together inside a service is covered in OpenTelemetry in depth.