Why architecture matters here
Architecture here is driven by three hard constraints that a naive single-key design cannot satisfy simultaneously. The first is blast radius. If one key encrypts a billion objects and that key is compromised, every object is compromised, and there is no partial containment. Envelope encryption lets each object — or each tenant, or each table — carry its own data key, so a leaked DEK exposes exactly one blob and nothing else. The root key, which could expose everything, never leaves the HSM and is never handled by application code, so the surface where it could leak is orders of magnitude smaller than where DEKs are used.
The second constraint is rotation without re-encryption. Compliance regimes and good hygiene demand that keys rotate on a schedule. If your one key directly encrypted petabytes, rotation would mean decrypting and re-encrypting petabytes — an operation so expensive that teams simply never rotate, which is how you end up with a five-year-old key protecting production. With envelope encryption, rotating the CMK is cheap: you only need to re-wrap the DEKs (kilobytes of key material), or even defer that and re-wrap lazily on next access, because the CMK's job is only to protect DEKs, not data. The expensive bulk ciphertext never has to move.
The third constraint is the performance and boundary mismatch between where keys must be protected and where encryption must happen. A KMS/HSM is a slow, network-bound, tightly guarded service; it can perform a few thousand cryptographic operations per second, and every call is audited. Bulk data encryption, by contrast, needs to run at gigabytes per second, locally, in the data path. You cannot send terabytes through the HSM. Envelope encryption threads this needle: the HSM does one tiny, auditable wrap operation per object (or per batch), and the fast local CPU — using AES-GCM, often hardware-accelerated with AES-NI — does the bulk work with the DEK. You get HSM-grade key protection and line-rate throughput at the same time, which is the whole reason the pattern exists.
The architecture: every piece explained
Top row: the participants. The application is any service that needs to protect data at rest — a storage layer, a database, a secrets vault. The KMS/HSM is the trust anchor: it holds the customer master key (CMK), a key that is generated inside the hardware boundary and can be used but never exported in plaintext. The CMK is identified by an ARN or key ID, governed by a key policy, and every use of it is logged. The data key (DEK) is a symmetric key — typically AES-256 — generated fresh for each object or logical partition. The envelope is the on-disk artifact: the ciphertext of your data plus the wrapped DEK plus metadata (which CMK wrapped it, the encryption context, the AAD, an algorithm identifier).
Middle row: the mechanics. GenerateDataKey is the pivotal KMS call. The application asks the KMS for a new data key under a named CMK; the KMS returns two forms of the same key — the plaintext DEK (for immediate local use) and the ciphertext DEK (the DEK encrypted under the CMK, safe to persist). The application encrypts its payload locally with the plaintext DEK using AES-GCM, an authenticated cipher that produces ciphertext plus an authentication tag so tampering is detectable. Crucially, the application then scrubs the plaintext DEK from memory and keeps only the wrapped form. The key hierarchy — CMK wraps DEK, DEK protects data — is what makes the whole chain of custody auditable and revocable at the top.
The decrypt path reverses this: the application reads the envelope, extracts the wrapped DEK, and calls Decrypt on the KMS to unwrap it. The KMS checks IAM authorization and the encryption context, unwraps the DEK inside the HSM, and returns the plaintext DEK. The application decrypts the payload locally with AES-GCM (verifying the auth tag), then scrubs the DEK again. Bottom rows: rotation generates a new CMK version and re-wraps DEKs on a schedule or lazily; audit and IAM ensure every wrap and unwrap is both authorized by a fine-grained policy and recorded in an immutable log; and the ops strip captures what actually keeps this running — key policies, grants, decrypt-result caching with short TTLs, cross-region key replication for disaster recovery, and a rehearsed break-glass procedure for when normal access paths fail.
End-to-end flow
Trace one object end to end. A user uploads a 40MB medical scan to a storage service. The service is configured with a CMK ARN for the healthcare tenant. On write, it calls GenerateDataKey(KeyId=cmk_health, KeySpec=AES_256, EncryptionContext={tenant:acme, purpose:phi}). The KMS authorizes the caller's IAM role against the CMK's key policy, generates a 256-bit DEK inside the HSM, encrypts that DEK under the CMK (binding the encryption context into the wrap so it must be presented again at decrypt), and returns both the plaintext and wrapped forms. The whole call takes a few milliseconds.
The service now encrypts the 40MB payload locally with AES-256-GCM using the plaintext DEK and a fresh random nonce, producing ciphertext and a 16-byte authentication tag at multiple gigabytes per second on AES-NI hardware. It assembles the envelope: ciphertext, nonce, auth tag, the wrapped DEK, the CMK ARN, and the encryption context. It writes that envelope to object storage. Then — the step teams forget — it overwrites the plaintext DEK buffer so it no longer lingers in the heap. The object is now at rest with no usable key stored anywhere near it; stealing the storage bucket yields only ciphertext and a wrapped key that is inert without a KMS call the attacker cannot make.
Later, a doctor requests the scan. The service reads the envelope, sees the wrapped DEK and CMK ARN, and calls Decrypt(CiphertextBlob=wrapped_dek, EncryptionContext={tenant:acme, purpose:phi}). The KMS re-checks IAM authorization, verifies the encryption context matches what was bound at wrap time (so a wrapped DEK stolen from one tenant cannot be unwrapped under another context), unwraps the DEK in the HSM, and returns the plaintext DEK. The service decrypts the payload with AES-GCM, and the auth-tag check confirms the ciphertext was not tampered with in storage. It streams the scan to the doctor and scrubs the DEK.
Now the interesting operational cases. Suppose the security team rotates cmk_health after a suspected employee incident. The new CMK version becomes the default for wrapping, but old envelopes still reference the old version — KMS keeps prior versions available for decrypt, so reads keep working while new writes use the new key; a background job can re-wrap old DEKs lazily. Suppose instead the team must revoke access entirely: they disable the CMK. Instantly, every Decrypt call fails, so every object under that key becomes unreadable cluster-wide without touching a single byte of ciphertext — crypto-shredding as a control-plane operation. And because the service caches decrypt results with a 60-second TTL, a burst of reads for a hot object hits the KMS once, not once per read, keeping both latency and KMS request costs sane under load.