Apache Ranger — Centralized Authorization for the Whole Hadoop Stack

Why it matters

Compliance is the single biggest driver of Ranger adoption. GDPR, HIPAA, PCI-DSS, and every internal security policy demand that access be centrally controlled and auditable. Ranger produces the audit trail these frameworks require, in one queryable location, with one consistent format across every Hadoop service.

Ranger also lets security teams own security policy without needing to be Hadoop experts. Policies are written in a web UI by security engineers; the Hadoop admins do not touch them. This separation of duties is itself a compliance requirement in many organizations.

The architecture

Ranger has three main components. Ranger Admin is a web service that hosts the policy database, the authoring UI, and the audit browser. Ranger plugins are libraries embedded into each Hadoop service (HDFS NameNode, HiveServer2, HBase Master, etc.) that fetch policies from Admin and enforce them locally on every RPC. Solr or HDFS serves as the audit sink where every access decision is logged.

Policies are written in a resource-based model: specify the resource (path, database, table, topic), the users or groups, the allowed accesses, and optional conditions like time of day or IP address. Policies can also be tag-based, pulling classifications from Atlas so that any resource tagged 'PII' gets special handling.

How it works end to end

Ranger Admin runs as a Java web application backed by a database (MySQL or Postgres). Security engineers author policies through the UI. Each Ranger plugin embedded in a Hadoop service periodically polls Admin (default every 30 seconds) to sync the latest policies.

When a client makes a request to a Hadoop service, the plugin is consulted before the operation executes. It evaluates its cached policy set against three things — the resource being touched, the identity making the request, and the access type being attempted — and returns allow or deny; a deny aborts the operation immediately. What that consultation mechanically is, however, is not the same in every service, and the difference is large enough to change what the permissions already on your cluster mean. The next section takes it apart.

Every evaluation, allow or deny, is logged asynchronously to the audit sink. The audit record includes the resource, the identity, the operation, the decision, and the policy that led to the decision. Solr indexes these records so operators can search them; HDFS storage keeps a longer archive for compliance retention.

Where the plugin actually hooks in, and why it differs per service

The sentence you will read in most introductions — that the Ranger plugin intercepts a request before the service's own authorization runs — is a useful first approximation and a bad mental model to keep. It is close enough for Hive, HBase, Kafka and YARN. It is wrong for HDFS, in a way that changes what every mode bit and ACL on the cluster means.

Every plugin has the same shape: a jar on the service's classpath, a ranger-<service>-security.xml pointing at Admin, and a policy cache on local disk. What varies is the extension point each service exposes, and therefore where in the request lifecycle the decision lands and what native machinery is left standing behind it.

HDFS - the enforcer is replaced, not layered on top

The NameNode exposes one pluggable authorization slot, dfs.namenode.inode.attributes.provider.class. Whatever class you name there supplies an access enforcer that the NameNode calls instead of its built-in permission check, not after it. The Ranger HDFS plugin is such a class. The consequence people get wrong follows directly: a Ranger allow policy grants access to a path whose mode bits and ACLs would refuse it, because those bits are no longer the thing being evaluated. Ranger is not a filter narrowing what POSIX already permits. On that path it is what permits.

The corollary is that getfacl stops being a complete answer to "who can read this". The inode still carries its owner, group, mode and ACL entries; they are simply not consulted unless the plugin is configured to fall back to them when no policy matches, which is a single boolean in its configuration. HDFS Permissions and ACLs works that flag, the fallback semantics and the resulting auditing problem in full, and is the reference for the HDFS side; this article deliberately does not repeat it.

Hive, HBase, Kafka and YARN - one authorizer slot, checked before the operation

The other services also hand Ranger a single authorizer slot, but the picture is cleaner because there is no independent, persistent, per-object permission model still sitting underneath it.

HiveServer2 takes an authorizer through hive.security.authorization.manager. Ranger's implementation runs during query compilation, after the parser and the metastore have resolved the statement into concrete read and write entities, so the check sees the exact databases, tables and columns the plan will touch rather than the text the user typed. That placement is what makes column masking and row filtering possible at all, and it is why HiveServer2 is the trust boundary for Hive rather than Tez or LLAP.

HBase loads Ranger as a coprocessor on the Master and the RegionServers, so the decision happens inside the observer hooks ahead of the get, scan or mutation. Kafka takes a class implementing the broker authorizer interface, invoked per produce, fetch and metadata request. YARN takes an authorization provider that answers queue submission and administration questions. In each case Ranger occupies the slot the service's own grant-based mechanism would have used, so you are running one engine rather than two.

What survives here is stale native state rather than live native evaluation, and that is its own trap. An HBase cluster that used the built-in access controller before Ranger still has rows in its ACL table. A metastore that once used SQL standard authorization still has grants recorded in it. Neither is being consulted, both are still readable, and both will mislead whoever audits the cluster next.

Policies are pulled and cached, which is what keeps the cluster up

Nothing in the request path talks to Ranger Admin. Each plugin runs a background thread that polls Admin's REST endpoint on an interval — ranger.plugin.<service>.policy.pollIntervalMs, thirty seconds by default — sending the version of the policy set it currently holds. Admin returns the full set if the version has moved and a not-modified response if it has not, so a steady-state cluster of several hundred plugins costs Admin almost nothing.

Each version the plugin receives is written to a JSON file under ranger.plugin.<service>.policy.cache.dir. That file is the reason a Ranger outage is not a cluster outage. If Admin goes down, or its database does, every plugin keeps enforcing the last set it pulled; if the service itself restarts while Admin is still down, it loads the cached file from disk and comes up enforcing rather than failing open or failing shut. Losing Ranger costs you the ability to change policy and to browse audits. It does not cost you access control.

The price is a staleness window the length of the poll interval, and the risk in it is asymmetric. Granting access late is an annoyance: someone waits half a minute and retries. Revoking access late is a security event. When you delete a policy or remove a group membership, that change is not in force for up to a poll interval on every plugin independently, and the plugins are not synchronised with each other, so during that window some services enforce the new policy and some the old. If a revocation genuinely matters, revoke the Kerberos principal as well — that takes effect at the next ticket rather than the next poll — and treat the Ranger edit as cleanup rather than as the control.

Shortening the interval trades that window against load on Admin and its database, which is the component that buckles first on large clusters. Ten seconds across two thousand plugins is a very different query rate from thirty.

The policy model - resources, allow, deny, and the two kinds of exception

A resource-based policy names a service, a resource, and a set of rules. The resource is typed and hierarchical rather than a flat string: HDFS policies name a path with a recursive flag; Hive policies name database, table and column as separate levels, each accepting wildcards; HBase policies name table, column family and column qualifier; Kafka policies name a topic or a consumer group. Wildcards apply within a level, so a Hive policy covering the column ssn in every table of the sales database is one policy, not one per table.

Each policy carries up to four rule lists, and the two that get forgotten are the useful ones. Allow conditions list the users, groups and roles that receive the listed accesses. Exclude from allow conditions subtracts from that set, so "everyone in analysts except these three contractor accounts" is a single policy rather than an allow plus a compensating deny. Deny conditions refuse explicitly and are evaluated ahead of allows. Exclude from deny conditions subtracts from the deny set, so "nobody reads this table except the on-call rota" is again one policy.

Deny beating allow at the same priority is exactly what makes the exclusion lists necessary. Without them, any broad deny has to be shattered into narrower denies to let one account through, and that shattering is what rots over eighteen months until nobody can say what the policy set actually expresses. Ranger also evaluates tag-based policies ahead of resource-based ones, so a tag deny short-circuits before any path or table policy is examined at all.

Priority and override - the semantics people misread

Every policy carries a priority, normal or override, and the misreading is predictable: people assume override means "this policy wins", full stop. What it actually changes is evaluation order. Override-priority policies are considered first, and within each priority tier deny still beats allow. The practical consequence is the surprising one — an override allow can defeat a normal-priority deny, which inverts the "deny always wins" instinct that everybody carries over from firewall rules and IAM.

That is the feature behaving as designed. Override priority exists so a platform team can carve a legitimate exception out of a blanket restriction — a backup service account that must read a table the compliance deny policy covers — without editing the compliance policy and without weakening it for anyone else. Used that way it is a precise instrument. Used as a convenient way to unstick a failing job, it punches a silent hole in a control that a reviewer reading only the deny policy will believe is intact.

The discipline that keeps this honest is simple and rarely applied: override policies should be rare enough that you can enumerate them from memory, and each one should say in its description what it overrides and why. A cluster with thirty override policies has no describable access model, and the deny policies on the screen are decoration.

Tag-based policies and Atlas classifications

Resource-based policies scale with the number of resources, which is the wrong axis to scale on. A rule such as "columns holding national identifiers are readable only by the privacy team" written against resources becomes one policy per table, forever, and every newly created table lands unprotected by default because nobody has written its policy yet.

Tag-based policies invert that relationship. Apache Atlas holds classifications on entities — a Hive column classified PII, an HDFS path classified RESTRICTED. A Ranger component called tagsync subscribes to Atlas change notifications and maintains a tag store that Admin serves to plugins alongside the resource policies. You then write a small number of policies about tags instead of a large number about resources: deny read on PII to everyone outside one group, mask anything classified PCI.

Three properties make the extra moving parts worth it. The policy count stops growing with the warehouse. A newly created column inherits protection the moment it is classified — including by an automated classifier that never sleeps — rather than when a human notices it exists. And a classification can span services, so the rule that governs a Hive table can also govern the HDFS path beneath it when both plugins resolve the same tagged entity, which is precisely the gap that per-resource rules leave open.

The failure mode is the new dependency you have taken on. Tag policies are only as fresh as tagsync and only as complete as the classification behind them. An unclassified PII column is not denied by a tag policy; it is invisible to it, and invisible is indistinguishable from allowed. Tag-based rules are a scaling layer on top of resource-based ones, not a replacement for the coarse resource policies that fence off a database wholesale.

Row filters and column masks, and the bypass nobody plans for

Because the Hive check happens during compilation against resolved entities, Ranger can return more than a yes or a no: it can hand back a predicate and a set of column expressions that HiveServer2 folds into the plan. A row-level filter policy attaches a boolean expression to a table for a given group — something on the order of region = 'EMEA' — and HS2 conjoins it into every statement that group runs against that table. A masking policy attaches a transformation to a column: nullify it, hash it, show only the last few characters, show only the year of a date, or apply a custom expression.

Two consequences matter more than the feature list. The first is that filters and masks are query rewrites, so aggregates honour them. A user under a row filter gets a count over their visible rows and not the table's true count, and nothing in the result set signals the difference. Two teams under different filters will reconcile numbers with each other, disagree, and open a data-quality ticket for something that is working correctly; expect that support load and document the filters somewhere the analysts can read.

The second is where the per-service hook difference comes back to bite. Masking and filtering live inside HiveServer2. A job that reads the table's underlying files directly — a Spark application pointed at the warehouse path, a copy tool, anything that skips the SQL layer — is authorized by the HDFS plugin against the path, and the HDFS plugin knows nothing about column masks. If the path is readable, the raw column is readable. Column-level control in Hive is therefore only as strong as your ability to keep readers out of the storage layer, which in practice means the warehouse directories must be denied to exactly the users you are masking for, and the SQL endpoint must be the only supported way in. Teams that deploy masking without closing the filesystem path have bought an audit finding, not a control.

Users and groups - where "the policy looks right but access is denied" comes from

Ranger runs a separate daemon, usersync, which imports users and groups on a schedule from a source: the local Unix accounts of a host, or far more commonly LDAP or Active Directory. What it produces is the list the Admin UI offers you when you type a group name into a policy. It is a directory of names, for authoring.

The classic incident follows from the fact that this is not necessarily the list consulted at enforcement time. The identity that reaches a plugin is a short username, produced by Kerberos authentication and the auth_to_local rules; Hadoop Kerberos is the reference for that path. It is worth being precise here, because the two get blurred constantly: Ranger performs no authentication whatsoever. It is handed an identity that Kerberos has already established and decides only what that identity may do. If authentication is broken, Ranger has nothing to say about it. The group list attached to that username has historically been produced by the service's own group mapping — the NameNode's shell or LDAP lookup, for example — which is a completely separate piece of configuration from the one usersync uses.

So the mismatch takes several shapes and they all produce the same symptom, a policy that reads correctly and denies anyway. Usersync imported finance from the directory, but the NameNode resolves groups by shelling out to the operating system, that host is not joined to the directory, and the user resolves to no groups at all. Or the directory returns Finance and the policy says finance, and matching is exact. Or usersync imported the account as alice@CORP.EXAMPLE.COM while auth_to_local produced alice, so the policy names a principal that never arrives. Or the group was granted in the directory twenty minutes ago and neither the usersync cycle nor the service-side group cache has expired yet.

Diagnosis is always the same move, and it is worth making it reflexively before reading the policy a fourth time: find out what username and what group list the enforcement point actually saw. The audit record for the denied request carries the username; the service's own group-resolution tooling gives you the group list it computed. Nine times in ten the policy was fine and the group list was empty. Rewriting the policy against individual usernames makes the symptom vanish and the access model unmaintainable, which is exactly why it is the popular fix.

The audit pipeline and its sinks

Every decision a plugin makes, allow and deny alike, becomes an audit record: timestamp, user, client address, resource, access type, the result, and the identifier of the policy that produced it. That last field is what makes the trail worth keeping, because it answers "why was this permitted" rather than only "this was permitted", and a compliance review that cannot trace a grant back to a policy is not a review.

Writing is asynchronous and buffered inside the plugin's own process. Records land on an in-memory queue, a batch writer drains it to the configured destinations, and if a destination is unreachable the batch spills to a local spool directory to be replayed later. This is deliberate: the audit path must never add latency to a NameNode RPC or a region scan, and must never fail a request because a search index is down.

Two sinks matter in practice. Solr — SolrCloud in any real deployment — is the searchable one and backs the audit browser in the Admin UI; retention there is short by design, weeks rather than years, because audit volume on a busy cluster is enormous. HDFS is the archival one, written as dated files, cheap to keep for whatever retention your auditors demand and painful to query without a table defined over it. Most sites run both, and the useful third step is shipping onward to a SIEM so that Ranger decisions correlate with everything else in the estate.

The hazards are all about volume. Auditing every allow on a busy HDFS cluster produces a record per RPC, which is a firehose rather than a log. The Solr collection is undersized on its first day and stays that way until it starts rolling records early. And when a sink is unreachable for long enough, the spool directory quietly fills the local disk on every node running a plugin, converting a monitoring outage into a service outage. Size the spool, alarm on its depth, and decide deliberately whether allows need the same fidelity as denies.

Ranger KMS is a separate service, not a plugin

Ranger KMS does not follow the plugin model at all. It is a standalone key management service — a derivative of Hadoop KMS with a backing database and Ranger policy enforcement over key operations — sitting in the HDFS transparent-encryption path, answering requests to generate and decrypt encrypted data encryption keys. Its policies are authored in the same Admin UI and audited through the same pipeline, which is the entire reason it exists as a Ranger component rather than as a separate product.

The property worth having is the separation of duties it makes possible: permission to read a file and permission to decrypt the key that file is encrypted under are two different authorizations held in two different policy sets, so an HDFS superuser who can traverse an encryption zone still retrieves ciphertext unless a KMS policy grants the decrypt operation. HDFS Encryption Zones is the reference for the encryption mechanics and the key hierarchy; treat Ranger KMS as the authorization and audit layer bolted onto them.

Operationally it is a hard dependency in a way that Ranger Admin is not. Plugins survive Admin being down because they cached the policies. Nothing caches a decrypted key. If KMS is unavailable, reads and writes inside encryption zones fail outright, so it needs its own high availability, its own capacity planning and its own alerting, sized and monitored separately from Admin.

High availability, component by component

The components have genuinely different availability requirements, and it pays to be explicit about them rather than treating "Ranger" as one thing on the architecture diagram.

Admin is a stateless web application over a shared database. Run two or more instances behind a load balancer, point the plugins at the virtual address, and it is covered; the real single point of failure is the database, which needs whatever replication your organisation already runs for anything else it cares about. Solr should be SolrCloud with replicas, sized against the audit rate rather than the policy count, because the audit rate is three or four orders of magnitude larger.

Usersync and tagsync are the awkward pair. Both are schedule-driven writers, and running two active copies means duplicated work against the directory and against Atlas. Deploy them as singletons with a fast path to restarting them elsewhere, and accept that an hour of downtime means an hour of stale group memberships and classifications rather than an outage.

Then the part that is easy to lose in a design review: the enforcement plane has no availability requirement on the control plane whatsoever. Plugins hold their policies locally, so Admin, its database and Solr all down at once degrades you to "cannot change policy, cannot browse audits, spool directories filling" — serious, time-bounded, and not a data-access outage. Alerting should be designed around that shape rather than around a single Ranger health check that pages at three in the morning for something the cluster is surviving.

Failure modes worth rehearsing

The plugin starts with nothing. A service restarts while Admin is unreachable and the cache directory is empty, unwritable, or was wiped by a host rebuild. What happens next is service-specific, and you want to know the answer for yours in advance rather than during. On HDFS it collapses onto the fallback flag; elsewhere an authorizer holding no policies denies.

The plugin was never installed on one node. Rolling a plugin out is a per-service, per-host configuration change, and the one RegionServer or the one HiveServer2 instance that missed it enforces nothing while reporting healthy. Verify by testing a known-denied access against every endpoint, not by reading the configuration management run log.

The recursive policy written at the wrong level. An HDFS policy on the root path with the recursive flag set, written to unblock one team, grants across the namespace. Ranger will let you save it and the UI will not flinch.

Policy count outruns evaluation. Every request evaluates against the policy set for its service. Several thousand wildcarded resource policies on a service under heavy RPC load shows up as latency in the service, not in Ranger, which makes it a genuinely confusing first diagnosis. This is the strongest practical argument for moving repetitive rules onto tags.

The audit trail has a hole exactly where you need it. Spool overflow, an undersized Solr collection rolling records early, or retention left at a default rather than set to the compliance requirement. All three are discovered during an investigation, which is the worst available time to discover them.

Two authorization systems, one of them forgotten. Leftover HDFS ACLs on a cluster running with native fallback enabled, orphaned rows in the HBase ACL table, stale metastore grants. None of these appear anywhere in the Ranger UI, and the Ranger UI is where everyone looks.