Why architecture matters here
Secrets architecture matters because credential leaks are one of the most common breach vectors. Getting secrets right — no long-lived credentials, no secrets in git, verified workload identity, audit everything — dramatically reduces attack surface.
Cost is small — managed secrets services are cheap.
Reliability comes from HA secrets managers + graceful degradation. When Vault is down, apps can't start; ensure HA + reasonable caching.
The architecture: every piece explained
Walk the diagram top to bottom.
Workload. Application needing DB creds, API keys, etc.
Secrets Manager. Vault, AWS SM, GCP Secret Manager. Central store + API.
Storage backend. Encrypted at rest; often replicated for HA.
Workload identity. SPIFFE, cloud IAM roles, Kubernetes ServiceAccounts. Prove who you are without long-lived secrets.
Dynamic short-lived. Vault dynamic secrets: create DB user on demand with TTL; auto-revoke.
Rotation. Auto-rotate every N days; apps refresh transparently.
Audit log. Every access logged: identity + secret + timestamp.
Envelope encryption. Data encrypted with data key; data key encrypted with KMS. Prevents storage-level exposure.
CI/CD integration. Pipelines get secrets via OIDC (GitHub Actions to AWS role) or Vault agent. No secrets in git.
Break-glass. Rare emergency access; heavily audited.
End-to-end secret flow
Trace a workload flow. Kubernetes pod starts. Its ServiceAccount is bound via Vault Kubernetes auth.
Pod's init container: sends SA token to Vault; receives Vault token with policy allowing db_creds/mysql/reader.
App queries Vault: db_creds/mysql/reader. Vault creates fresh DB user; TTL 1 hour; returns username + password.
App connects to DB. Uses ephemeral creds.
After 45 min, Vault agent refreshes creds. New user created; old revoked after grace period.
Audit log records every secret access with pod identity.
Static secret example: API key for Stripe. Stored in Vault; rotated quarterly by ops team. App reads on start.
CI/CD: GitHub Actions builds. Uses OIDC token to AssumeRole in AWS → temporary AWS creds. No long-lived AWS keys in git.
Emergency: break-glass access granted for 30 min to specific engineer. Audit shows: incident ID + engineer + secret accessed.