App SDK + Collector pattern
Apps emit OTLP via SDK to a local OTel Collector (sidecar or DaemonSet). Collector batches, retries, transforms, exports to one or more backends. Putting collectors close to apps lets you decouple SDK shipping from backend choice.
Tail-based vs head-based sampling
Head sampling: decide at span start (random, e.g., 10%). Cheap, but misses interesting tail traces (errors, slow requests). Tail sampling: collect all spans, decide at trace completion (keep all error traces + slow traces + N% of normal). Right answer; needs a tail-sampling collector tier.
Tiered collector deployment
Agent collectors (close to apps): batch + basic enrichment. Gateway collectors (a few in central locations): tail sampling, attribute reshaping, cost gating. Backend (Tempo, Honeycomb, etc.) receives the filtered stream. ~10x cost reduction vs sending everything direct.