Why architecture matters here
Logs fail on cost and shape. Unstructured text is a disaster to query. High-cardinality fields balloon storage. Missing redaction leaks PII. Retention too short loses evidence during incidents.
The architecture matters because each layer has knobs: shipper filters, schema conventions, tiered storage, retention policy.
The architecture: every piece explained
The top strip is the pipeline. Application emits structured log lines (JSON). Shipper — Fluent Bit, Vector — collects on the node. Aggregator — Kafka or OTel Collector — routes at scale. Storage is Elastic, OpenSearch, or Loki.
The middle row is the shape. Schema convention — mandatory fields (timestamp, level, service, trace_id). Correlation via trace_id + user_id joins logs with traces + metrics. Redaction scrubs PII at the shipper. Retention tiers keep hot for search, cold for compliance.
The lower rows are consumption. Query + visualize via Kibana or Grafana. Alerts on log-based conditions. Ops handles cost, volume, and governance.
End-to-end flow
End-to-end: service logs JSON per request with trace_id, user_id, latency. Shipper adds host, container, and env fields. Aggregator batches to storage. Hot retention 7 days; warm 30 days; cold 1 year (S3). Query looks up all logs for a trace_id during an incident. PII redacted at shipper. Alert fires when error_rate spikes. Cost dashboard shows storage per team.