Plain-text logs are search-only. Structured logs (JSON) are queryable. The difference shows up the first time you need to count errors by user_id, or trace a request across 5 services. Doing structured logging well requires three things: a consistent schema, trace correlation, and PII discipline.
Schema first
Define your log schema once: timestamp, level, service, trace_id, user_id, message, attributes. Every service logs the same shape. Don't let teams invent fields ad-hoc — you'll regret it when the SIEM ingests inconsistent data.
Trace correlation
Every log line includes trace_id and span_id from OTel context. In Loki/Datadog/Splunk, you can click a log → see the full trace, or click a trace span → see all logs for that span. This bidirectional pivot is the unlock for incident debugging.
PII scrubbing
Email, SSN, credit card, phone, IP — these MUST NOT land in logs unredacted. Use a logging middleware that scrubs known field names BEFORE serialization. Periodically audit your log streams with a PII scanner (e.g., AWS Macie, GCP DLP) for what slipped through.
Log levels that mean something
| Level | Meaning | Production rate |
|---|---|---|
| ERROR | Page someone | < 0.1% |
| WARN | Investigate within hours | < 1% |
| INFO | Audit / SLO measurement | ~10% |
| DEBUG | Off in prod | 0% |
Don't log secrets
Tokens, passwords, encryption keys — NEVER. Even at DEBUG. Once logs leave your system (S3, SIEM), they're radioactive. A simple regex middleware blocking Authorization headers and password fields catches 95% of accidents.