HDFS Encryption Zones — Transparent Per-File Encryption via KMS

Why it matters

Encryption at rest is table stakes for enterprise HDFS. Regulators expect it, security teams require it, and customer contracts often mandate it. Without HDFS Encryption Zones, the only way to encrypt data was full-disk encryption at the DataNode level, which is coarse-grained, does not survive backup workflows, and does not protect against attackers who compromise the OS but not the disks.

Zone-level encryption is finer grained: different sensitivity levels can live in different zones with different key rotation schedules, different KMS backends, and different access policies. This gives security teams the flexibility to match encryption to actual risk instead of applying a blanket policy.

The architecture

An encryption zone is an HDFS directory declared as encrypted. Every file created under an encryption zone gets its own Data Encryption Key generated at file creation time. The DEK encrypts the file's bytes on the client using AES-CTR. The DEK itself is then wrapped by the Encryption Zone Key, and the wrapped DEK is stored as an extended attribute on the file's inode.

The EZ Key never leaves the KMS. When a client wants to decrypt a file, it fetches the file's wrapped DEK from HDFS, sends it to KMS along with proof of authorization, and receives the unwrapped DEK back. All decryption happens client-side using the unwrapped DEK.

How it works end to end

Creating a new file in an encryption zone works like this: the client asks the NameNode to create the file, the NameNode takes an already-wrapped DEK from the pool it keeps topped up from the KMS, stores that wrapped DEK as an xattr on the inode, and hands the client the same wrapped copy. The client calls the KMS itself to have it unwrapped, then uses the resulting plaintext DEK to encrypt bytes locally before streaming them to DataNodes, so the DataNodes only ever see ciphertext and the NameNode never holds usable key material.

Reading is symmetric: the client fetches file metadata including the wrapped DEK, sends the wrapped DEK to KMS asking for it to be unwrapped, and receives the plaintext DEK back. The client then reads ciphertext from DataNodes and decrypts locally.

Key rotation is straightforward for the EZ Key: a new version is generated in KMS. Existing files continue to work because their wrapped DEKs reference the old EZ Key version. New files are wrapped with the new EZ Key version. Optional re-encryption walks the zone and updates every wrapped DEK to the newest EZ Key version.

What encryption at rest here is actually defending against

HDFS transparent data encryption answers one question well. If somebody carries a disk out of the datacentre, or pulls a decommissioned drive out of a skip, or gains root on a DataNode and reads block files straight off the local ext4 mount, what do they get? With an encryption zone in place they get ciphertext and nothing else. Block files under the DataNode data directories hold exactly the bytes the client sent, and the client sent them already encrypted. There is no key material in the block file, none in the adjacent metadata file, and nothing in the DataNode process that would help.

The second case is the platform administrator with root on the hosts and the hdfs superuser identity. That person can list any path, read any block file, and even extract the wrapped key material through the raw view of the namespace. What they cannot do is turn wrapped key material into a usable key, because unwrapping is a call to the KMS and the KMS enforces its own authorisation list. That is the whole separation-of-duties argument, and it rests on one organisational fact rather than any configuration: the KMS has to be administered by people who are not the HDFS administrators. If one person holds both the cluster superuser and edit rights on the KMS access lists, this feature buys nothing against that person and no property in any XML file will change it.

The list of things it does not cover is longer

Metadata is plaintext. Path names, directory structure, file sizes, ownership and timestamps live in the fsimage and edit log, none of which is encrypted. A path like /finance/pii/2026/q3/ssn_export_uk.parquet leaks a great deal before anyone reads a single byte.

Anyone legitimately authorised. A user who passes the HDFS permission check and appears in the KMS list gets the plaintext key, full stop. Their Spark or MapReduce task, running as them in a YARN container, holds that key in heap memory for the life of the stream. Compromise the job and you have the data. This is not a control against your own users or against the code they are permitted to run.

Intermediate data. Shuffle spills, container local directories, task logs, and anything an engine writes to node-local scratch sit entirely outside HDFS and are entirely unencrypted unless you separately configured that engine's own spill encryption. A job that reads a zone and spills a join to local disk has undone the guarantee for the duration of the job, on every worker.

Anything requiring integrity. More on that below, but the short version is that the cipher mode in use gives confidentiality and does not give tamper evidence.

The zone is a directory, and the rules around it are unusually strict

A zone is declared on an HDFS directory, and that directory must be empty at the moment you declare it. There is no operation that encrypts data in place. Existing files do not become encrypted because you drew a zone around their parent, and there is no offline conversion tool. The migration is always the same shape: create a new empty directory, declare it a zone, copy the data in, verify, delete the original. Budget for two copies of the dataset on disk during the cutover, and think about what happens to the plaintext blocks left on the DataNodes after the delete, because the blocks are only actually gone once the deletion has propagated and the DataNodes have removed the files.

Once declared, the zone covers everything beneath it at any depth. Subdirectories are not zones of their own; they simply inherit. Every file created anywhere under the path gets a fresh per-file key wrapped by that one zone key.

Zones do not nest. You cannot declare a zone inside an existing zone, which means the granularity of your key policy is fixed by the granularity of your top-level directory layout. If finance and HR need different keys with different access lists, they must be sibling zones, not two children of one zone. Getting this wrong is expensive to fix, because fixing it means copying the data again.

You cannot move a directory into a zone, out of one, or between two of them with a rename. HDFS rejects it outright. That restriction looks arbitrary until you look at where the keys live, which is the next section.

Three keys, one xattr, and a NameNode that holds nothing useful

There are exactly three pieces of key material in play, and keeping them straight makes every other behaviour in this article obvious.

The zone key is a named key that lives in the KMS. It is versioned, it never leaves the KMS in plaintext, and it is what your compliance auditor thinks of as "the key". One zone, one named key.

The data encryption key is generated fresh for a single file. It is what actually encrypts that file's bytes. It is never written to disk in plaintext anywhere, and it exists in plaintext only inside the memory of a client that is currently reading or writing that file.

The wrapped data encryption key is that per-file key encrypted under one specific version of the zone key. This is what HDFS persists, as an extended attribute on the file's inode named raw.hdfs.crypto.file.encryption.info, together with the initialisation vector and the cipher suite identifier. The zone directory itself carries a matching attribute naming the key it was created with.

So the NameNode's job is to be a courier. It stores wrapped key material, hands it to clients along with the rest of the file's metadata, and never possesses a plaintext key for any file it serves. A memory dump of the NameNode, a stolen fsimage, or a leaked edit log yields wrapped keys and no way to unwrap them.

There is one asymmetry worth internalising because it drives capacity planning. On the create side, the NameNode keeps a pool of pre-generated wrapped keys fetched from the KMS in advance, refilled by background threads once the pool falls below a low-water mark. A file creation therefore does not block on a KMS round trip, and a burst of creates does not become a burst of KMS requests. On the read side there is no equivalent: the client presents the wrapped key on open and waits for the KMS to answer. Writes are buffered against KMS latency; reads are not.

The generic wrap-and-unwrap pattern underneath all of this is the standard one. If you want the vendor-neutral model - key hierarchies, versions, hardware protection boundaries, crypto-shredding as a control plane operation - it is covered in Cloud KMS architecture and KMS envelope encryption. Everything below is specific to the HDFS implementation and its Hadoop KMS.

Counter mode, seeking, and the tamper evidence you do not get

The cipher suite is AES in counter mode with no padding, and the choice is forced by how Hadoop reads files. A reader that wants byte 700,000,000 of a two-gigabyte file must be able to jump directly to the block holding it and begin decrypting there, because that is precisely what a split-per-block execution model does on every task in every job. Counter mode allows it: the keystream at any offset is derived from the initialisation vector and a counter computed from that offset, so decryption is random-access and stateless. It also means the ciphertext is exactly the same length as the plaintext, which keeps block boundaries and file lengths unchanged.

An authenticated mode such as GCM would force sequential processing of a whole stream to validate its tag, which would break splittability outright. So the trade is deliberate, and the consequence is that the cipher gives confidentiality and no integrity at all. There is no authentication tag. The stored CRC32C checksums still catch bit rot, a bad disk, or a truncated write, but a checksum is not a message authentication code. An attacker who can write to a DataNode's local filesystem can modify ciphertext and recompute the checksum to match, and the reading client will decrypt the result without complaint.

Counter mode is also malleable in a structured way: flipping a bit in the ciphertext flips exactly the corresponding bit in the decrypted plaintext, with no avalanche. An attacker who knows the file format can make targeted edits without ever knowing the key. If your threat model includes an active adversary with write access to block storage rather than only a passive one who steals disks, this feature is not the control you want and you need integrity at a layer above it.

The KMS is a separate service with its own access model

Hadoop KMS is a standalone web service, not a NameNode subsystem. Clients find it through a provider URI set as hadoop.security.key.provider.path in core-site.xml, and both the NameNode and every client need that setting to be correct and reachable.

Authentication is Kerberos, and this matters more than it first appears. Every access rule in the KMS is expressed in terms of user and group names. Without Kerberos the identity a client presents is an unverified string that anyone can set, so every rule in the file becomes decorative. Deploying encryption zones on a simple-authentication cluster is theatre: the data is encrypted, and any user on the network can ask politely for the key and receive it.

Authorisation lives in kms-acls.xml and has two layers that are both evaluated. Cluster-wide entries, hadoop.kms.acl.<OPERATION>, decide who may perform an operation at all. Per-key entries, key.acl.<keyname>.<OPERATION>, narrow that down to one named key. The operations that carry the weight for encryption zones are:

The shape you want in production falls straight out of that table. The NameNode principal gets generate and nothing else, so a compromised NameNode can mint new wrapped keys but cannot read a single existing file. Analyst groups get decrypt on the specific keys of the zones they are entitled to, never on a wildcard. The security team holds management. The HDFS administrators appear in none of the lists at all, and that absence is the control.

The access file is re-read when it changes, so edits take effect without restarting the service. That cuts both ways: a good change lands immediately and so does a bad one, with no deployment gate in between. Treat that file as production configuration under review, not as something an operator edits at three in the morning.

Running the KMS so it does not become the cluster's weakest link

Every open of an encrypted file is a KMS request. Not every read - the key is unwrapped once and held for the lifetime of that input stream - but every open. A job that touches fifty thousand files issues fifty thousand unwrap requests, and they do not arrive spread evenly across the job. They arrive in a wall at task start, because every task opens its inputs in its first second. The number to size against is peak concurrent task starts multiplied by files opened per task, and the honest way to get it is to measure your existing job mix rather than to guess.

The failure signature when you have undersized it is distinctive and easy to misdiagnose: jobs that used to reach their first record in seconds now sit for minutes with every task blocked, cluster CPU is idle, HDFS looks perfectly healthy on every dashboard, and the only place the truth is visible is the KMS request log. People chase the NameNode for a long time before they think to look.

Running more than one instance is mandatory in any deployment that matters, and it comes with three requirements that are easy to get subtly wrong.

The key store has to be genuinely shared. A Java keystore file copied onto each host is the classic trap. It works in testing and then, the first time somebody rolls a key, one instance serves the new version and the others do not, and reads fail on a fraction of requests depending on which instance the load balancer picked. Production deployments back the KMS with a shared database or a hardware module - Ranger KMS is the usual choice in Hadoop distributions - precisely so that all instances see one authoritative copy.

Delegation token signing secrets have to be shared too. If each instance signs with its own secret, a token minted by one is rejected by another, and behind a round-robin load balancer that produces intermittent authentication failures that scale with load and vanish when you test by hand. The standard answer is a ZooKeeper-backed secret provider so all instances sign and verify with the same material.

Transport security is not optional here. Unwrapped keys travel back to the client over this connection. An HTTP endpoint hands plaintext key material to anything on the network path and makes the entire scheme pointless while still passing the compliance checkbox that says data is encrypted at rest.

Zone boundaries - rename, copy, and distcp

Renaming inside a zone is an ordinary metadata operation and stays cheap. Renaming across a boundary is refused, and the reason is mechanical rather than a policy decision. The file's bytes are encrypted under a key wrapped by the source zone's key, and its extended attribute names that key. Moving the inode into a different zone would produce a file whose stored key material refers to a key the destination zone does not use, and there is no way to reconcile that without rewriting every byte. HDFS refuses rather than creating a file it cannot later read. The same logic applies to moving plaintext in from outside: unencrypted blocks cannot be adopted into a zone by a metadata operation.

So crossing a boundary always means copy and delete: read, decrypt, re-encrypt under the destination key, write. For a single file the shell handles it transparently. For bulk movement you reach for distcp, and distcp has two modes here that behave very differently.

Ordinary distcp, and why -skipcrccheck appears

A normal distcp out of a zone decrypts on read and re-encrypts on write under whatever the destination requires. The bytes stored at the destination are therefore legitimately different from the bytes stored at the source, so the block checksums differ, so distcp's post-copy comparison fails on every single file. Adding -skipcrccheck silences that, and it is the correct flag, but be clear about what you gave up: the copy is no longer verified end to end, and a truncated or corrupted transfer will not be caught by the tool. Compensate with a file count and byte count reconciliation, or an application-level hash of the decrypted content, rather than declaring the migration done because the command exited zero.

Copying through the raw namespace

The other mode prefixes both source and destination paths with /.reserved/raw and preserves extended attributes. Now distcp moves ciphertext byte for byte and carries each file's wrapped key along with it. Nothing is decrypted, no unwrap requests hit the KMS at all, the stored checksums match on both sides so the normal verification works, and the transfer is faster because no cipher runs anywhere.

The catch is that the destination is only usable if the same zone key exists there, under the same name, in a KMS the destination cluster can reach. That makes raw-path copying exactly right for replicating a zone to a disaster-recovery cluster that shares key infrastructure, and exactly wrong for anything else. It also requires superuser, because the raw view deliberately exposes wrapped key material that ordinary permission checks were never designed to guard.

Trash and snapshots inside a zone

Deleting a file normally moves it into the user's trash directory under their home path, which is a rename. Rename across a zone boundary is forbidden. Those two facts collide, and HDFS resolves the collision by giving each zone its own trash directory at the root of the zone, so a deleted file stays inside the boundary and stays encrypted rather than being silently relocated to a plaintext home directory.

Two operational consequences follow. Deleted-but-not-yet-expired data keeps consuming the zone's space, so a zone does not shrink at delete time and capacity investigations need to account for it. And emptying trash is now a per-zone chore rather than a per-user one, which surprises people who have automated the home-directory version. The checkpoint and expiry mechanics themselves are unchanged and are covered in HDFS Trash.

Snapshots interact more subtly. A snapshot preserves the inode state including the wrapped-key attribute, which is what makes restoring a file from a snapshot inside a zone work at all: you recover key material that still unwraps correctly. But a snapshot is immutable by construction, so nothing can ever rewrite the wrapped key captured inside it. The zone key version that attribute refers to must therefore stay available in the KMS for as long as the snapshot exists. Retire that version and the snapshot silently becomes an unreadable artifact that still consumes space and still appears in listings. In practice this means key-version retention policy and snapshot retention policy are one policy, owned by one team, or you will eventually delete a key version and discover the loss months later during a restore drill.

Rolling the key versus re-encrypting the zone

These are two different operations and conflating them is the most common misunderstanding in this whole feature.

Rolling creates a new version of the zone key. From that moment, newly created files get their per-file keys wrapped under the new version. Nothing that already exists changes in any way. Every existing file still carries a wrapped key bound to an older version, and every one of those older versions has to remain available in the KMS or those files stop opening. Rolling on its own buys forward protection and nothing retroactive.

Re-encrypting the zone walks the namespace under the zone and rewrites each file's wrapped key so it is bound to the current version. It runs on the NameNode in batches, reports progress, and can be cancelled and resumed. The crucial property is in what it does not touch: it re-encrypts keys, not data. The per-file key protecting each file's bytes is unchanged, the ciphertext on disk is never read or rewritten, and no block moves. That is exactly why it is affordable on a petabyte-scale zone - the work is proportional to file count, not to data volume, and it is a metadata operation from the DataNodes' point of view.

The security consequence follows directly and is worth stating plainly. If you are rotating for hygiene, for a compliance schedule, or to retire an old version so you can finally stop keeping it alive, re-encryption does precisely what you need. If you are rotating because a per-file key leaked - captured out of a client's memory, printed into a debug log, exfiltrated from a compromised container - re-encryption does nothing for you at all, because that leaked key still decrypts the same unchanged ciphertext. The only remedy in that case is to rewrite the affected data under new per-file keys, which means copying every file. Know which situation you are in before you start.

And remember the snapshot constraint from the previous section: even after a re-encryption reports complete, you cannot retire the old key version until every snapshot that captured a wrapped key under it has been deleted.

Where the CPU cost actually lands

Encryption and decryption both run in the client, which means the cost lands on whichever host is executing the task, not on the storage layer. On a current x86 core with AES instruction support, counter-mode AES is fast enough that it is almost never the limiting factor for a job that is already moving data across a disk or a network link.

The configuration detail that decides whether you get that speed is whether Hadoop is genuinely loading its native cipher implementation. If the native library is absent, built without the right support, or simply not on the library path, Hadoop falls back to the pure-Java provider. It does this quietly. There is no error, no warning at job submission, and no metric that says "you are now decrypting the slow way" - only a fleet of tasks that are inexplicably CPU-bound with cipher frames at the top of every stack sample. Verify the native path at deployment time on every node type, including the ones somebody added last month.

Two smaller costs are easy to forget. Short-circuit local reads still function inside a zone, because the DataNode is still just passing a file descriptor and the client is still reading blocks directly - the cipher work is simply added on top in the reading process, so you keep the benefit and pay the CPU (see short-circuit reads). And the unwrap request adds latency to every open. For a job reading a few large files that is invisible. For a job opening millions of tiny ones it is a second tax stacked on an access pattern that was already the wrong shape (see the small files problem).

The failure modes that actually happen

KMS unavailable means the data is unreadable

This is the one that wakes people up. The KMS sits on the critical path of every open, so an outage there is a total data outage across every zone while HDFS itself remains perfectly healthy. The NameNode keeps serving metadata, listings work, quota and usage commands answer normally, replication continues - and every read of encrypted content fails. Creation also fails once the NameNode's pre-fetched pool of wrapped keys drains, which happens some minutes into the incident rather than immediately.

The confusing part is that streams already open keep working, because their key is already in memory. Long-running jobs sail on while newly submitted ones fail instantly, so the first reports look intermittent and workload-specific. Monitor the KMS as a tier-one dependency with the same seriousness as the NameNode, alert on its error rate and latency rather than only on whether the process is alive, and make sure the on-call runbook names it explicitly, because it is not where anyone's instinct sends them first.

Losing the zone key is permanent

If the KMS backing store is lost and there is no restorable backup, every file in every zone under those keys is unreadable forever. There is no recovery path, no vendor escalation, and no forensic option. The key store needs its own backup schedule, its own restore rehearsal, and its own retention policy - and once you have that backup, it is the single most sensitive artifact your organisation owns and needs to be guarded accordingly.

The mirror image of this is a genuine feature. Destroying a zone key is the fastest and most complete way to render a dataset unrecoverable without touching a single byte of it, which makes it a legitimate tool for decommissioning, for tenant offboarding, and for satisfying a deletion obligation across a dataset too large to rewrite. Just make sure the deliberate case and the accident case cannot be confused with each other at the console.

Misconfigured access lists fail in two different directions

The HDFS permission check and the KMS check are independent and both must pass, which produces two symptoms that look nothing alike. A user with filesystem read permission but no unwrap entry does not get garbage bytes and does not get a permission denial on the path - they get an authorisation failure raised from the key provider, which surfaces as a stack trace mentioning the KMS in the middle of a job. Support desks consistently misroute this one, so document it. A user with an unwrap entry but no filesystem permission never gets far enough to ask, because they cannot read the file's attributes in the first place.

The dangerous misconfiguration is neither of those, because it does not produce a symptom. A wildcard left in a cluster-wide entry grants unwrap on every key in the cluster, and nothing anywhere will tell you, because from HDFS's point of view every permission check passed exactly as it should. Audit the key lists as configuration, review them the way you review firewall rules, and never let a wildcard survive a code review. The related quiet failure is a zone created with a key that nobody except its creator can unwrap - always verify the access list before you spend a week loading data into the zone.

Setting one up end to end

The sequence below is the shape of a real deployment, with the ownership boundary in the right place: step one belongs to whoever runs the KMS, and steps two onward belong to whoever runs HDFS.

# 1. Create the zone key. Run as the KMS administrator, NOT the HDFS admin -
#    keeping these two roles separate is the entire point of the feature.
hadoop key create finance_pii_key -size 256
hadoop key list -metadata

# 2. Create an EMPTY directory and declare it a zone. It cannot already
#    contain data, and it cannot sit inside another zone.
hdfs dfs -mkdir -p /data/finance_pii
hdfs crypto -createZone -keyName finance_pii_key -path /data/finance_pii
hdfs crypto -listZones

# 3. Load data. Crossing the boundary is a real copy, never a rename, and the
#    checksums cannot match because the stored bytes are legitimately different.
hadoop distcp -update -skipcrccheck /staging/finance /data/finance_pii

# 4. Replicate the zone to DR without decrypting anything. Both sides go
#    through the raw namespace, xattrs are preserved, checksums DO match here,
#    and the destination KMS must already hold the same named key.
hadoop distcp -px \
    hdfs://prod-nn/.reserved/raw/data/finance_pii \
    hdfs://dr-nn/.reserved/raw/data/finance_pii

# 5. Rotate. Rolling only affects new files; re-encryption re-wraps the
#    existing per-file keys and does not touch a single block of data.
hadoop key roll finance_pii_key
hdfs crypto -reencryptZone -start -path /data/finance_pii
hdfs crypto -listReencryptionStatus

Two things to check afterwards that the commands will not tell you. First, read a file back as an ordinary member of the intended group, not as the superuser, because the superuser's success proves nothing about whether the unwrap list is right. Second, read a file as the hdfs superuser and confirm that it fails - if it succeeds, your key access list still includes the HDFS administrators and the separation you thought you had does not exist.