Why architecture matters here
The collector exists because the alternative — applications exporting directly to backends — welds policy into every deployed binary. Change vendors, add a destination, tighten PII redaction, adjust sampling: each becomes a coordinated redeploy of every service in the company. The collector inverts the coupling: applications speak OTLP to a local endpoint and are done; everything that will change — destinations, credentials, sampling rates, enrichment, cost controls — lives in pipeline configuration owned by the observability team. It is the same argument as message brokers and API gateways: an indirection layer at exactly the seam where change is constant.
The economics run deeper than convenience. Telemetry is a firehose priced by the drop: traces at production volume can cost more than the compute they observe, and the highest-leverage controls — tail-based sampling (keep every error trace and latency outlier, 1% of the boring rest), metric aggregation and filtering, log routing by value tier (errors to the expensive indexed store, debug to cheap object storage) — are only implementable in a layer that sees complete signals before the vendor's meter runs. A gateway fleet with tail sampling routinely cuts trace spend 10× while improving incident coverage, because the 1% it keeps is chosen, not random.
And the collector is where compliance meets telemetry physically. Attributes carry PII (URLs with emails, headers with tokens) that must not reach third-party backends; data-residency rules may require EU telemetry to stay in EU backends; audit requires knowing what left. OTTL-based redaction, attribute allowlists, and region-aware routing in the gateway tier are enforceable, auditable policy — versus hoping every team's instrumentation remembered. The collector is not plumbing; it is the policy engine for the most privacy-dense data stream your infrastructure produces.
The architecture: every piece explained
Top row: the component model. Receivers ingest: OTLP gRPC/HTTP (the native protocol from SDKs and other collectors), a full Prometheus scraper (the collector can replace Prometheus agents), filelog tailing with parsers, Kafka, cloud-provider receivers, and dozens more from the contrib registry. Processors run in declared order per pipeline: memory_limiter first (refuse data before OOM), then transforms/filters/enrichment, with batch last-ish (compression efficiency for export). Exporters deliver: OTLP to vendors or upstream collectors, Prometheus remote-write, Kafka, file/debug — and a pipeline may fan out to several (the migration pattern: dual-write old and new vendor for a quarter). Pipelines wire it per signal type; the same receiver can feed multiple pipelines, and connectors join them: the spanmetrics connector consumes the trace pipeline and emits RED metrics into the metrics pipeline — service dashboards derived from traces with zero extra instrumentation.
Middle row: topology and the hard feature. Agent mode puts a collector on every node (daemonset) or beside every pod (sidecar): apps export over localhost (no network loss window), and the agent enriches with what only locality knows — k8s metadata (pod/namespace/node via the k8sattributes processor), host resources. Gateway mode is the central fleet behind a load balancer: fewer, bigger collectors owning policy that requires the full stream. Tail sampling is the gateway's marquee job and its scaling constraint: to decide 'keep this trace?' on error/latency criteria you must buffer complete traces — which means all spans of a trace must reach the same gateway instance. The pattern is two-layer: a first tier load-balances by trace_id (the loadbalancing exporter), a second tier buffers (seconds of spans in memory) and applies policies (error traces: always; p99 latency: always; the rest: probabilistic 1%). Memory limiter and sending queues are the self-protection kit: soft/hard memory thresholds that trigger refusal (backpressuring sources) before the kernel's OOM killer makes the decision, and per-exporter persistent queues that ride out backend outages.
Bottom rows: the transformation language and the ops reality. OTTL (the OpenTelemetry Transformation Language) powers the transform processor: redact (replace_pattern(attributes["url"], ...)), enrich, rename, drop attributes, reshape spans/metrics/logs — the policy vocabulary, applied declaratively. The ops strip is recursive but essential: the collector exports its own telemetry (accepted/refused/dropped per component, queue depths, memory), and the observability team's first dashboard is the pipeline's; config ships as code through CI with validation; capacity is planned per signal per tier.
End-to-end flow
Follow telemetry through a two-tier estate during a normal day and a bad one. A checkout pod emits spans via OTLP to its node's agent (localhost, 1ms). The agent's pipeline: memory_limiter → k8sattributes (annotates pod, deployment, node) → resource detection → batch → OTLP exporter to the gateway VIP. Metrics and logs ride parallel pipelines on the same agent — the Prometheus receiver scrapes the pod's endpoints; filelog tails container logs with parsing to structured records.
At the gateway, tier one hashes each span's trace_id via the loadbalancing exporter to tier two, where the tail_sampling processor buffers traces for 10 seconds and applies policy: this trace touched an error? Keep. Duration > 2s? Keep. Contains a span from the payments service (compliance requirement)? Keep. Otherwise: 1% lottery. Kept traces flow through an OTTL transform that redacts card-number-shaped strings and strips two internal-only attributes, then export: the APM vendor (primary) and a Kafka topic feeding the data lake (long-term). The spanmetrics connector meanwhile emits per-service RED metrics from all spans (pre-sampling — the metrics see 100% even though only 1.4% of traces persist), which is how dashboards stay accurate over sampled traces. Net effect on the bill: 98.6% fewer trace ingests, zero lost error visibility.
The bad day: the APM vendor's ingest has a 40-minute outage. Gateway exporters hit failures; persistent sending queues absorb — disk-backed, sized for an hour at peak — while the Kafka export continues unaffected (independent queues per exporter). Queue-depth alerts fire (the pipeline's own telemetry); nobody loses data; when the vendor recovers, queues drain with rate limiting so the backlog doesn't DDoS the recovered ingest. Two weeks later, the same machinery earns its keep differently: the company adds a second APM vendor for evaluation — one exporter block and a 5% routing rule in the gateway config repo, reviewed, merged, live in an afternoon; not one application team was involved. And when security asks 'can any telemetry contain auth tokens?', the answer is an OTTL rule and its test suite in git — a pull request, not an archaeology project across three hundred services.